Question: exists.Register(plugins) noderestriction.Register(plugins) nodetaint.Register(plugins) <del> label.Register(plugins) // DEPRECATED, future PVs should not rely on labels for zone topology </del> podnodeselector.Register(plugins) podtolerationrestriction.Register(plugins) runtimeclass.Register(plugins)

In 1.30, the PersistentVolumeLabel admission plugin blocks creation of in-tree AzureDisk and vSphere PVs: The reason is that the cloud provider was removed (), but the admission plugin still calls GetCloudProvider there, which fails: The PV should be admitted + created. Enable PersistentVolumeLabel admission plugin. For example, when running local-up- Create AzureDisk PV on any 1.30 cluster, you don't need Azure cloud: /sig storage /priority important-soon <detailsToday's master </details<detailsAzure </detailsNo response <detailslocal-up- </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am fixing the worst bug in and I plan to backport it to 1.30. However, we should consider removing the whole PersistentVolumeLabel admission plugin. Most of the cloud providers were removed, only GCE remains and IMO even that one will be removed sooner or later. The admission plugin has no way to get volume zone + region and is therefore useless. The admission said (in 1.26, where we had all in-tree cloud providers): Such webhook never materialized for AWS, Azure nor vSphere and I am not aware of any plans for GCE.
It did materialize in So removing it from k/k is the best way forward.


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "context" "fmt" "time" storagev1 "k8s.io/api/storage/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" types "k8s.io/apimachinery/pkg/types" "k8s.io/client-go/util/retry" "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e/storage/utils" admissionapi "k8s.io/pod-security-admission/api" "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" ) var _ = utils.SIGDescribe("StorageClasses", func() { f := framework.NewDefaultFramework("csi-storageclass") f.NamespacePodSecurityLevel = admissionapi.LevelBaseline ginkgo.Describe("CSI Conformance", func() { ginkgo.It("should run through the lifecycle of a StorageClass", func(ctx context.Context) { scClient := f.ClientSet.StorageV1().StorageClasses() var initialSC, replacementSC *storagev1.StorageClass initialSC = &storagev1.StorageClass{ TypeMeta: metav1.TypeMeta{ Kind: "StorageClass", }, ObjectMeta: metav1.ObjectMeta{ GenerateName: "e2e-", }, Provisioner: "e2e-fake-provisioner", } ginkgo.By("Creating a StorageClass") createdStorageClass, err := scClient.Create(ctx, initialSC, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create the requested StorageClass") ginkgo.By(fmt.Sprintf("Get StorageClass %q", createdStorageClass.Name)) retrievedStorageClass, err := scClient.Get(ctx, createdStorageClass.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get StorageClass %q", createdStorageClass.Name) ginkgo.By(fmt.Sprintf("Patching the StorageClass %q", retrievedStorageClass.Name)) payload := "{"metadata":{"labels":{"" + retrievedStorageClass.Name + "":"patched"}}}" patchedStorageClass, err := scClient.Patch(ctx, retrievedStorageClass.Name, types.StrategicMergePatchType, []byte(payload), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch StorageClass %q", retrievedStorageClass.Name) gomega.Expect(patchedStorageClass.Labels).To(gomega.HaveKeyWithValue(patchedStorageClass.Name, "patched"), "checking that patched label has been applied") ginkgo.By(fmt.Sprintf("Delete StorageClass %q", patchedStorageClass.Name)) err = scClient.Delete(ctx, patchedStorageClass.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete StorageClass %q", patchedStorageClass.Name) ginkgo.By(fmt.Sprintf("Confirm deletion of StorageClass %q", patchedStorageClass.Name)) scSelector := labels.Set{patchedStorageClass.Name: "patched"}.AsSelector().String() type state struct { StorageClasses []storagev1.StorageClass } err = framework.Gomega().Eventually(ctx, framework.HandleRetry(func(ctx context.Context) (*state, error) { scList, err := scClient.List(ctx, metav1.ListOptions{LabelSelector: scSelector}) if err != nil { return nil, fmt.Errorf("failed to list StorageClass: %w", err) } return &state{ StorageClasses: scList.Items, }, nil })).WithTimeout(30 * time.Second).Should(framework.MakeMatcher(func(s *state) (func() string, error) { if len(s.StorageClasses) == 0 { return nil, nil } return func() string { return fmt.Sprintf("expected StorageClass to be deleted, found %q", s.StorageClasses[0].Name) }, nil })) framework.ExpectNoError(err, "timeout while waiting to confirm StorageClass %q deletion", patchedStorageClass.Name) ginkgo.By("Create a replacement StorageClass") replacementSC = &storagev1.StorageClass{ TypeMeta: metav1.TypeMeta{ Kind: "StorageClass", }, ObjectMeta: metav1.ObjectMeta{ GenerateName: "e2e-v2-", }, Provisioner: "e2e-fake-provisioner", } replacementStorageClass, err := scClient.Create(ctx, replacementSC, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create replacement StorageClass") ginkgo.By(fmt.Sprintf("Updating StorageClass %q", replacementStorageClass.Name)) var updatedStorageClass *storagev1.StorageClass err = retry.RetryOnConflict(retry.DefaultRetry, func() error { sc, err := scClient.Get(ctx, replacementStorageClass.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "unable to get Storage %q", replacementStorageClass.Name) sc.Labels = map[string]string{replacementStorageClass.Name: "updated"} updatedStorageClass, err = scClient.Update(ctx, sc, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to update StorageClass %q", replacementStorageClass.Name) gomega.Expect(updatedStorageClass.Labels).To(gomega.HaveKeyWithValue(replacementStorageClass.Name, "updated"), "checking that updated label has been applied") scSelector = labels.Set{replacementStorageClass.Name: "updated"}.AsSelector().String() ginkgo.By(fmt.Sprintf("Listing all StorageClass with the labelSelector: %q", scSelector)) scList, err := scClient.List(ctx, metav1.ListOptions{LabelSelector: scSelector}) framework.ExpectNoError(err, "failed to list StorageClasses with the labelSelector: %q", scSelector) gomega.Expect(scList.Items).To(gomega.HaveLen(1)) ginkgo.By(fmt.Sprintf("Deleting StorageClass %q via DeleteCollection", updatedStorageClass.Name)) err = scClient.DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: scSelector}) framework.ExpectNoError(err, "failed to delete StorageClass %q", updatedStorageClass.Name) ginkgo.By(fmt.Sprintf("Confirm deletion of StorageClass %q", updatedStorageClass.Name)) err = framework.Gomega().Eventually(ctx, framework.HandleRetry(func(ctx context.Context) (*state, error) { scList, err := scClient.List(ctx, metav1.ListOptions{LabelSelector: scSelector}) if err != nil { return nil, fmt.Errorf("failed to list StorageClass: %w", err) } return &state{ StorageClasses: scList.Items, }, nil })).WithTimeout(30 * time.Second).Should(framework.MakeMatcher(func(s *state) (func() string, error) { if len(s.StorageClasses) == 0 { return nil, nil } return func() string { return fmt.Sprintf("expected StorageClass to be deleted, found %q", s.StorageClasses[0].Name) }, nil })) framework.ExpectNoError(err, "timeout while waiting to confirm StorageClass %q deletion", updatedStorageClass.Name) }) }) }) </ins>

[X] test approval issue : [X] test pr : [X] two weeks soak end date : 22 Sept 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three StorageClass endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following StorageClass endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/sig storage
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } switch { case waiting > 0: <ins> glog.V(5).Infof("pod waiting > 0, pending") </ins> // One or more containers has not been started return api.PodPending case running > 0 && unknown == 0:

I have a cluster with ~130 pods. Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything. thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 127 Running 1 STATUS 6 Unknown thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 82 Running 1 STATUS 51 Unknown Nothing changed in between these calls.
cc I blame the pod cache.
has recently changed pod cache? FWIW, I also observed this behavior: 19 pods Running in one call and all Unknown in the next. With 0.10.0.
Yes, I think PR on pod cache should fix the problem reported here. Also it should fix the issue reported by about adds-on services. I have a pending PR to have kubelet deciding on PodStatus too.
Could this have caused ?
I don't see how. I'll comment on that issue.
I don't think this could be the cause of that one.
I don't think PR on pod cache will fix this. That one clears the node status. This issue is with pod going from Running to Unknown even when nodes keep running just fine.
Totally consistent observation based on my personal experience. I just assumed we were shooting for some kind of Hesinberg Uncertainty Principle. S , Tim Hockin wrote:
Agree with clears node status only. One possibility is that we hit transient network problem in between the calls that fails health check, so controller set node condition to not reachable, hence unknown pods. I'll make this more robust anyway. Other than that, it's hard to think of a reason that node status results in the flakiness.
In the absence of node information, the pod status defaults to Unknown. Maybe there's another source of this specific problem, but the pod cache is a source of continuous problems, partly because it's not just a cache, but is implementing or duplicating a significant chunk of system logic in a half-baked manner. My goal is to eliminate it -- see .

Short of that, we need to at least implement , eliminate nodes from the pod cache, and implement a real node cache.
I can look into this. Repro info from Tim, who said he was running on a one-node cluster but has seen the problem with multi-node clusters also: run-container hostnames --image=kubernetes/servehostname --replicas=1 --port=9376 --labels=app=hostnames expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" resize --replicas=150 rc hostnames but I had to hack the node a bit to make it get past 100 on the node: sysctl -w fs.inotify.maxuserinstances=8192 echo "DOCKERNOFILE=" /etc/default/docker; /etc/init.d/docker restart (fails sometimes, dunno why)
I can verify that the situation is rather bad. Lots of fluctuation in the numbers of Pending, Running, and Unknown. serve_hostname containers on the node are exiting; I'm not sure why, or if the pod itself is terminating or just the container. Lots of the following in the log (not sure what this means yet): Sat, 07 Feb 2015 00:06:22 +0000 hostnames-ecd2r Pod failedScheduling {scheduler } Binding rejected: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
The pause container is also exiting, so I'm guessing that means the whole pod is exiting, not just serve_hostname...
Ugh, there is an etcd size limit you might be hitting too, I forgot. If you are seeing the fluctuations, maybe you got far enough, though. On Feb 6, 2015 5:13 PM, "David Oppenheimer" wrote:
Just a few more observations: This is reproducible (and a little less chaotic) with one node and 100 pods (but not with 50 pods, so it's definitely load-dependent). You can get to 100 Running pods (0 Pending or Unknown), and then a minute later a dozen or so will become Unknown, and then later go back to Running, etc. As mentioned earlier, the containers continue running on the node all this time (sudo docker ps wc -l), so it is definitely some kind of reporting problem.

As an aside, it looks like there is some kind of race condition with IP address allocation, as during startup (but not at steady state) a number of pods exit with the following in their log (sudo docker logs xxx): listen tcp :9376: bind: address already in use They do restart OK (I'm not sure if this is being restarted locally on the node or making a round-trip back to the scheduler).
Almost certainly the restart is just kubelet retrying after some time. On Feb 8, 2015 2:57 PM, "David Oppenheimer" wrote:
I found that if I comment out the line with the arrow next to it (see below) in , the Pods stop oscillating between Running and Unknown. Commenting out this line changes the logic from "if get an error from Kubelet, change pod status to Unknown" to "if get an error from Kubelet, keep pod status unchanged." I saw in the master logs intermittent errors of this form, which was causing the pods to flip back to Unknown: E0209 04:19:17. 4996 ] Error getting info for pod default/hostnames-v5drx: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-v5drx This seems to correspond to lines like this in the kubelet logs: I0209 06:38:11. 4704 ] GET /stats/default/hostnames-gfw5j/-b026-11e4-a070-/hostnames?num_stats=33: (218.857218ms) 500 goroutine 730 [running]: [ ... stack trace, but they're not always the same; I'll dig into this some more, the default stack trace depth wasn't going back past the various functions that just serve up the error ... ] However, pods still flip between Running and Pending, even though no containers are exiting on the minion, so there's another problem here too.
Thanks for investigating this! Pods really shouldn't flip between Running and Pending, as I believe that the only way that they go Pending is if Pod.Status.Host is empty, and this really shouldn't happen post scheduling, unless we are somehow dropping information. Maybe we need to make sure that we don't wipe out status on the error case?

--brendan , David Oppenheimer
I also investigated the same issue last Friday. I observed exact the same as what described above, except the related kubelet log should be: Meanwhile, I observed master is trying to query pod / and failed all the time: Talked to offline, he is going to look into why running pods go back to Pending; meanwhile, I am going to figure out Unknown cases here.
That would be great. It is very hard for me to make sense of any of the scalability experiments that I am doing while the pod status is in hysteresis.
Just to summarize: (1) the Running<-Unknown flapping is caused by the line of code I mentioned earlier; (2) the reason that line of code is being hit on the master is due to two errors on the kubelet (the one I identified earlier was wrong; Dawn found these): I0209 06:42:09. 4704 ] GET /api/v1beta1/podInfo?podID=hostnames-v5drx&podNamespace=default: (8.82543048s) 500 E0209 04:34:13. 4996 ] Error getting info for pod default/hostnames-2y8ny: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-2y8ny (3) we don't yet know the cause of the Running<-Pending flapping; I'm going to look into that now
(and Dawn is looking into why we're hitting those two errors on Kubelet)
Are we serving requests concurrently in Kubelet? (Is it possible not to when using go's servemux?) I don't see any locking around the pod map accesses.
This was marked for the fixit but Dawn and I are working on it. Assigned to myself.
I am also investing time writing code to try and report what is going on with unstable cluster status. It would be good to know what you and Dawn are doing in case I am wasting my time. Thanks. , David Oppenheimer
This particular problem is easily reproduced manually using the instructions Tim listed, so I'm not sure it's necessary to write anything sophisticated to drive the cluster into this state.

Anyway, maybe give me and Dawn a couple of days and if we can't figure this out by then we can call in the cavalry.
Am I wrong, or are these instructions also within bounds of an e2e test? Just because our existing e2es only do one or two things doesn't mean we can't do more within O(minutes), assuming you're dealing with the same image.
The "flips back to pending" problem appears to be because updatePodStatus() writes to the cache but not back to the Pod's status. computePodStatus() initializes newStatus based on pod.Status, not based on the cached value of status. This second behavior seems wrong in light of the first behavior. However, I believe this only causes a problem in combination with my fix described earlier, which as a result doesn't actually do what I described (it doesn't "keep the status as-is" but rather copies it out of the pod's status; to "keep the status as-is" it should be copying it out of the cache). I can see if this fixes things. (We also should fix the root cause stuff on the node side, which Dawn is working on.)
Assign the issue to me ... I didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. The proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this.
Dawn, my suggestion is that computePodStatus() should return the current cached value whenever it gets an error from Kubelet. I can implement this if you want.
, why shouldn't updatePodStatus simply write through to the pod's status? I'd think either we always read the cached value first, or we always write through to the pod and read from there. Only reading the cached value upon an error seems counter-intuitive to me...
why do you computePodStatus returns a cached value here?

The only place calls computePodStatus is updatePodStatus, which triggered by two operations: 1) periodically update all containers 2) the client asks for PodStatus, and there is no cached one, instead calling computePodStatus to retrieve one. I think a cleaner way is 1) Kubelet handles all the errors except this transient one. Of course, we need to make sure docker is auto-restart and there is no too many open files error on node side too. But they are separate issues and should be handled by node side through script or configure etc. 2) Now back to master side, if node status is ok, kubelet should return something; if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round. I have one working PR to avoid the flips between Running and Unknown, but with increasing number of Pending from node side caused by docker failure. Still investigating it...
I didn't understand "if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round." If you're talking about the caching in the podCache on the master, then what you wrote is exactly what I was suggesting. (If you're talking about some kind of caching on the Kubelet, then that's different, because I am not aware of cache for status on the Kubelet.) I think always writing back to the pod's status might be fine but I don't know why the podCache was designed the way it was, so I didn't want to change that without asking Daniel first (I think he's the one who created it). I wasn't suggesting to only read the cached value upon an error; I meant always read the cached value under all circumstances.
and here is my understanding. Correct me if I am wrong. UpdateAllContainers periodically updates all containers, but it gets the initial pods from etcd through this line . each from , UpdateAllContainers calls passes to . computes and returns a new status. Note that the status is computed based on , i.e., the information in etcd. updates the pod cache with the new status.

Note that the new status is not written to etcd. If you repeat the same steps again, you will start with the stale pod status in the etcd because does not check the cached value.
I have tested since last night. The flaps between Running, Pending and Unknown are gone for about 10 hour running 300 pods on 4 nodes except two scenario: 1) etcd issue: cat kube- | grep "status to unknown" I0212 20:18:30. ] node doesn't exist: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] <nil, setting pod hostnames-yuy8d status to unknown ... This happens a couple of times last 10 hours. We need to figure out why. 2) NodeStatus becomes UNREADY. It happens once last 10 hours, and pods status on that node become Unknown which is expected. I didn't figure out why NodeStatus was UNREADY then, but I heard there is a network earlier this morning.
up to you whether to reopen this issue or file new ones for new issues.
I've also seen the "All the given peers are not reachable" message (see my second entry in this issue). I think that was what was referring to when he said "there is an etcd size limit you might be hitting too" but I may be mis-interpreting.
That error is the etcd client's catch-all for "something went wrong" - it's actually really bad that it obscures errors. , David Oppenheimer
, Tim Hockin wrote:
and Yes, that error message is cache-all error message, which need to investigation. In my particular case, it shouldn't be etcd size limits at all since we have that cluster built and upgraded more than 10 hours before, and all 300 pods were scheduled back then. There is no new events generated either from kubelet or other components since all bound pods are indeed happily running on the nodes. No new writes to etcd except NodeStatus, which is very less frequent and not much data. The problem of etcd here occurs when master tries to read from etcd with an error.


Question: node_upgrade=true node_prereqs=false local_binaries=false <ins> env_os_distro=false </ins> while getopts ":MNPlh" opt; do case ${opt} in

During a manual upgrade test, described in , a kubelet failed to communicate via SSL to its master. This is currently blocking the 1.4 release. There is not a repro for this yet. The cluster was running 1.3, and was upgraded to 1.4 successfully, but upon downgrading to 1.3, the kubelets one one or more machines are not able to talk to the master. Current kubelet version is . (EDITED) The error when talking to the master is (lots like this): I think this means that the kubelet cannot verify the master's cert. I can't find the master's cert. Since there is not flag to kubelet to tell it where to find its config, I assume it is using the default location, which is . It seems wrong that clusters[0].cluster.certificate-authority-data is empty. That should have the cert for the master, right? I see this thing: but I don't understand what process generates , and if, presumably, from that template, why it isn't setting the env var CA_CERT.
in case this is related to kubelet bootstrap changes in 1.4.
Currently node is :
Correction. The node is running 1.4.0-beta.5
If the kubelet hasn't set the then it will not go through the bootstrap code path. Is the kubeconfig provided through the bootstrap flag or the kubeconfig flag?
Not set. For reference, args are:
Question: Could you please tell me why your nodes are running with GCI image at the first place? The test is starting with 1.3 release, which is using old debian-based containervm image, and I thought doesn't change the image at all (separate topic).
It does change to gci on gce with the default upgrade scripts, i brought this up on an email thread. I'm guessing something was dropped when we translated salt grains (containervm) to gci env vars.
What the default upgrade scripts you were referring to? I just double-checked with test cluster which was upgrade to 1.4 through , the node is stay with debian-based containervm, but Kubernetes version is bumped up to 1.4-beta.X
Ok, I am assigning this to to confirm.

If upgrade logic from 1.3 to 1.4 changes the underneath os, that is wrong. I strongly believe this is just a simple operational error.
My understanding is that ignores base images altogether. , Dawn Chen wrote:
Hmm
I've tried and the node image did get updated to GCI. Before upgrade to v1.4: After upgrade to v1.4: After downgrade to v1.3: /cc
helped me debug this. He found that the metadata for the instance did not include the CA_CERT in it. That seems wrong.
Renamed from "x509 failure for kubelet after upgrade test"
Yeah, that's the error you'd get if the proper ca isn't loaded.
Will we get this problem if we don't upgrade to GCI (i.e. we stay with CVM) in -N? I guess we can try it and find out.
Manual upgrades have been successful if we don't switch image types. , David Oppenheimer < :
re: If you don't change the underneath image during the upgrade, the issue shouldn't occur.


Question: ``` ### NFS <del> Kubernetes NFS volumes allow an existing NFS share to be made available to containers within a pod. </del> <ins> Kubernetes NFS volumes allow an existing NFS share to be made available to containers within a pod. </ins> <del> [The NFS Pod example](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/nfs/test.yaml) demonstrates how to specify the usage of an NFS volume within a pod. In this example one can see that a volumeMount called "myshare" is being mounted onto /var/www/html/mount-test in the container "testpd". The volume "myshare" is defined as type nfs, with the NFS server serving from 172.17.0.2 and exporting directory /tmp as the share. The mount being created in this example is not read only. </del> <ins> See the [NFS Pod examples](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/nfs/) section for more details. For example, [nfs-web-pod.yaml](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/examples/nfs/nfs-web-pod.yaml) demonstrates how to specify the usage of an NFS volume within a pod. In this example one can see that a `volumeMount` called "nfs" is being mounted onto `/var/www/html` in the container "web". The volume "nfs" is defined as type `nfs`, with the NFS server serving from `nfs-server.default.kube.local` and exporting directory `/` as the share. The mount being created in this example is not read only. </ins>

This page: Links to this YAML: Which doesn't exist. There are several examples now in the NFS folder:
Thanks for reporting! Always happy to get docs up-to-date, especially to reduce friction. Please report / open PRs for any others. Also, if the examples / explanations aren't clear, we're happy to address issues for that as well.


Question: github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE= github.com/dustin/go-humanize v1.0.0 h1:VSnTsYCnlFHaM2/igO1h6X3HA71jcobQuxemgkq4zYo= github.com/dustin/go-humanize v1.0.0/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk= <del> github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153 h1:yUdfgN0XgIJw7foRItutHYUIhlcKzcSf5vDpdhQAKTc= </del> github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= github.com/emicklei/go-restful v2.9.5+incompatible h1:spTtZBk5DYEvbxMVutUuTyh1Ao2r4iyvLdACqsl/Ljk=

What happened: upgrade requests to aggregated APIs (like websocket-based watch requests) on a kube-apiserver using --ssh-user hang What you expected to happen: Websocket-based watch requests work properly How to reproduce it (as minimally and precisely as possible): a kube-apiserver configured to use a node dialer to reach services/pods (e.g. with ) an aggregated API server (like the sample-apiserver) on a node other than the kube-apiserver a websocket-based watch against the aggregated API Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery /cc lavalamp
The code in question is here: handlingInfo.proxyRoundTripper is built here: and inherits the proxyDialerFn built in from the following construction chain: However, drops the custom dialer contained inside and from the returned round tripper. That means that upgrade requests fall back to use the default dialer, which cannot reach the aggregated API server
cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/assign would be good to resolve with the apiserver proxy work


Question: # REVISION provides a version number fo this image and all it's bundled # artifacts. It should start at zero for each LATEST_ETCD_VERSION and increment # for each revision of this image at that etcd version. <del> REVISION?=1 </del> <ins> REVISION?=2 </ins> # IMAGE_TAG Uniquely identifies k8s.gcr.io/etcd docker image with a tag of the form "<etcd-version>-<revision>". IMAGE_TAG=$(LATEST_ETCD_VERSION)-$(REVISION)

What happened: The latest release of the debian-base image, v2.1.3, contains a vulnerable version of the apt package. CVE-2020- impacts apt version 1.8.2.1. What you expected to happen: Image should be updated to use at least apt 1.8.2.2 to clear the vulnerability. How to reproduce it (as minimally and precisely as possible):
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kube-proxy is using debain as base image. /assign Let me check other components
checked the base image in and I seems that no new debian or ubuntu official image is built for this CVE ?
That is correct, the purpose of this issue is to get a new image built and released, one which contains a resolution for the CVE.
per comments of @ tianon in Will update it once debain releases it.


Question: }) // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for service with type clusterIP [LinuxOnly] [Flaky]", func() { </del> <ins> ginkgo.It("should have session affinity work for service with type clusterIP [LinuxOnly]", func() { </ins> svc := getServeHostnameService("affinity-clusterip") svc.Spec.Type = v1.ServiceTypeClusterIP execAffinityTestForNonLBService(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for service with type clusterIP [LinuxOnly] [Flaky]", func() { </del> <ins> ginkgo.It("should have session affinity timeout work for service with type clusterIP [LinuxOnly]", func() { svc := getServeHostnameService("affinity-clusterip-timeout") svc.Spec.Type = v1.ServiceTypeClusterIP execAffinityTestForSessionAffinityTimeout(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should be able to switch session affinity for service with type clusterIP [LinuxOnly]", func() { </ins> svc := getServeHostnameService("affinity-clusterip-transition") svc.Spec.Type = v1.ServiceTypeClusterIP execAffinityTestForNonLBServiceWithTransition(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should have session affinity work for NodePort service [LinuxOnly] [Flaky]", func() { </del> <ins> ginkgo.It("should have session affinity work for NodePort service [LinuxOnly]", func() { </ins> svc := getServeHostnameService("affinity-nodeport") svc.Spec.Type = v1.ServiceTypeNodePort execAffinityTestForNonLBService(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. <del> ginkgo.It("should be able to switch session affinity for NodePort service [LinuxOnly] [Flaky]", func() { </del> <ins> ginkgo.It("should have session affinity timeout work for NodePort service [LinuxOnly]", func() { svc := getServeHostnameService("affinity-nodeport-timeout") svc.Spec.Type = v1.ServiceTypeNodePort execAffinityTestForSessionAffinityTimeout(f, cs, svc) }) // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should be able to switch session affinity for NodePort service [LinuxOnly]", func() { </ins> svc := getServeHostnameService("affinity-nodeport-transition") svc.Spec.Type = v1.ServiceTypeNodePort execAffinityTestForNonLBServiceWithTransition(f, cs, svc)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: [sig-network] Services should have session affinity work for service with type clusterIP [LinuxOnly] [sig-network] Services should be able to switch session affinity for service with type clusterIP [LinuxOnly] [sig-network] Services should be able to switch session affinity for NodePort service [LinuxOnly] [sig-network] Services should have session affinity work for NodePort service [LinuxOnly] Testgrid link: Reason for failure: Dec 17 23:05:35.615: Connection to 10.0.207.245:80 timed out or not enough responses. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/sig testing
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I think this issue is resolved. my pipeline () passed and the issue that mentioned this one () has been merged. cc'ing Jordan just in case there is anything im missing. /cc
Until the root cause has been identified/fixed, or we see testgrid show this flake free for several days, we'll leave this open.
/assign these are four of the top-ten flakes according to if a fix is not identified shortly, we'll need to mark them as , which will move them out of the merge/release-blocking jobs
it seems that those errors are not happening when using KIND may it be a GCE environment issue?
Analyzing this occurence The tests fails timing out at 23:05:35.615 Dec 17 23:05:35.615: Connection to 10.0.207.245:80 timed out or not enough responses. The last update of the service port is at However, the endpoints are not present until 23:05:40. I1217 23:05:40. 1 ] Setting endpoints for "services-9616/affinity-clusterip-transition:" to [10.64.0.53:9376 10.64.1.114:9376] I1217 23:05:40.

1 ] sync-runner: 674.883613ms since last run, possible in 9.325116387s, scheduled in 9.325128618s During all that times it was logging That message belongs to the method func (hcs *server) SyncEndpoints(newEndpoints map[types.NamespacedName]int) error { do you have any clue why it takes so long to sync the endpoints or is just that the timeout should be raised?
Same happens in this one Dec 18 00:42:18.918: Connection to 10.0.60.18:80 timed out or not enough responses. endpoints are not ready until I1218 00:42:29. 1 ] Setting endpoints for "services-4166/affinity-clusterip-transition:" to [10.64.0.81:9376 10.64.1.71:9376 10.64.3.58:9376] I1218 00:42:29. 1 ] Setting endpoints for "services-4166/affinity-clusterip-transition:" to [10.64.1.71:9376 10.64.3.58:9376]
This is the constant that handles the timeout, however the comment says it should be enough time is it possible that there is a performance regression hidden somewhere?
The timeline shows that failures started around Dec 14th EDIT :man_facepalming: :smile: how could I forget this
That's when silent flake suppression/retry was turned off. I suspect this has been flaky for far longer than that.
Some interesting things, the apiserver is using the [::1] address. per example more entries in the log file I've opened an issue because I've observed that the apiserver mixing IP families caused retries and errors in kube-proxy
I'm starting to think that this may be a performance regression because the endpoints are not being updated in a timely manner, based on this analysis and the following entries and seems that these issues are related: and that has submited PR to fix this
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
four of the top ten flakes according to
any update on status of this?

if a fix is not imminent, these tests will be marked and will not run in blocking presubmit.
Taking a look
cc:
Finding so far: This error message should be redherring. It should not slow down or block kube-proxy iptables sync.
I opened to tag as [Flaky], I know these are hard to track down but they've been in the top 10 flakes for about a month now
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign
I've tried to reproduce it locally but I can't find entries for the iptables module, that AFAIK is the one that tracks the connections. I've created a deployment and a service and dumpled the iptables. After updating the service to use sessionAffinity and a timeout of 2 seconds, it seems that the iptables rules are correct: after several to the service for more than 2 seconds, it always hits the same backend. The problem is that I can't find entries in the /proc/net/xt_recent folder can it be something related to the iptables module and the switch to xtables?
Also, based on it seems that the test is not covering this scenario, it should be failing because it has to check that it switches the backend once the affinity times out
/assign
/cc
I couldn't find errors like this anymore and seems that the flakiness was reduced, maybe this problem was fixed by Analyzing one of the last occurrences it shows that the timeout is not enough In order to pass the test it should have 15 hits to the same backend in a row and during this time it was only able to get 13, there is a huge delay from the time the command is executed until the answer is received, i.e.

There is also another problem, in one of the occurrences the first 2 connections go to one backend and the other 13 goes to the another one it fails because it should have deleted the tracked hosts before retrying, otherwise is taking into account the wrong hosts to check teh affinity
ah, that was a problem before with the "affinity works at all" tests () but I had thought when I looked at this that the problem was that the affinity actually wasn't switching, not just that we weren't getting enough samples
also, suggested to me that there was actually a problem here that wasn't just "we aren't taking long enough to get 15 samples"
I think that this is another scenario we are not covering after 2 seconds we should get answers from a different backend, right?
:thinking: this is working to me from a pod ~I think that sessionAffinity only applies to pod-svc communication, that's why session affinity is not working in my example and in ~ my bad EDIT ok one mystery solved, scratch all previous theories, it seems that with a short time between requests the tracking entries never expire and it sticks forever, using 2 seconds between requests it gets requests from other backends
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.


Question: } } <ins> func newDaemonSetWithLabel(dsName, image string, label map[string]string) *appsv1.DaemonSet { return &appsv1.DaemonSet{ ObjectMeta: metav1.ObjectMeta{ Name:   dsName, Labels: label, }, Spec: appsv1.DaemonSetSpec{ Selector: &metav1.LabelSelector{ MatchLabels: label, }, Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: label, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "app", Image: image, Ports: []v1.ContainerPort{{ContainerPort: 9376}}, }, }, }, }, }, } } </ins> func listDaemonPods(c clientset.Interface, ns string, label map[string]string) *v1.PodList { selector := labels.Set(label).AsSelector() options := metav1.ListOptions{LabelSelector: selector.String()}

[x] test approval issue : [X] test pr : [x] two weeks soak end date : 7 June 2021 [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints which are untested. Create a Daemonset List all DaemonSets DeleteCollection of DaemonSetss Varify the DaemonSets have been deleted Using an existing conformance test as a template for a which validates that two new endpoints are hit. example endpoint | useragent -------------------------------------------|
createAppsV1NamespacedDaemonSet | (linux/amd64) kubernetes/$Format -- [sig-apps] Daemon set [Serial] should list and delete a collection of DaemonSets deleteAppsV1CollectionNamespacedDaemonSet | (linux/amd64) kubernetes/$Format -- [sig-apps] Daemon set [Serial] should list and delete a collection of DaemonSets listAppsV1DaemonSetForAllNamespaces | (linux/amd64) kubernetes/$Format -- [sig-apps] Daemon set [Serial] should list and delete a collection of DaemonSets listAppsV1NamespacedDaemonSet | (linux/amd64) kubernetes/$Format -- [sig-apps] Daemon set [Serial] should list and delete a collection of DaemonSets readAppsV1NamespacedDaemonSet | (linux/amd64) kubernetes/$Format -- [sig-apps] Daemon set [Serial] should list and delete a collection of DaemonSets (5 rows) If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
/triage accepted For Release 1.22
/open
/reopen
Reopened this issue. <detailsIn response to : /reopen </details


Question: }), "container_start_time_seconds": gstruct.MatchElements(containerID, gstruct.IgnoreExtras, gstruct.Elements{ <del> fmt.Sprintf("%s::%s::%s", f.Namespace.Name, pod0, "busybox-container"): boundedSample(time.Now().Add(-maxStatsAge).UnixNano(), time.Now().Add(2*time.Minute).UnixNano()), fmt.Sprintf("%s::%s::%s", f.Namespace.Name, pod1, "busybox-container"): boundedSample(time.Now().Add(-maxStatsAge).UnixNano(), time.Now().Add(2*time.Minute).UnixNano()), </del> <ins> fmt.Sprintf("%s::%s::%s", f.Namespace.Name, pod0, "busybox-container"): boundedSample(time.Now().Add(-maxStatsAge).Unix(), time.Now().Add(2*time.Minute).Unix()), fmt.Sprintf("%s::%s::%s", f.Namespace.Name, pod1, "busybox-container"): boundedSample(time.Now().Add(-maxStatsAge).Unix(), time.Now().Add(2*time.Minute).Unix()), </ins> }), "pod_cpu_usage_seconds_total": gstruct.MatchElements(podID, gstruct.IgnoreExtras, gstruct.Elements{

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI -- (containerd) (crio) [sig-node] ResourceMetricsAPI [NodeFeature:ResourceMetrics] when querying /resource/metrics should report resource usage through the resource metrics api Ever since this commit was merged (containerd) (crio) We need to see what the commit is breaking this particular NodeFeatures test on both crio and containerd.
/sig node
/cc
/cc
/triage accepted
ah, let's use the RT's issue /close
Closing this issue. <detailsIn response to : ah, let's use the RT's issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: true, }, { <ins> "Same spec, the label is different, the former doesn't have pod-template-hash label, same number of labels", generatePodTemplateSpec("foo", "foo-node", map[string]string{}, map[string]string{"something": "else"}), generatePodTemplateSpec("foo", "foo-node", map[string]string{}, map[string]string{extensions.DefaultDeploymentUniqueLabelKey: "value-2"}), false, }, { "Same spec, the label is different, the latter doesn't have pod-template-hash label, same number of labels", generatePodTemplateSpec("foo", "foo-node", map[string]string{}, map[string]string{extensions.DefaultDeploymentUniqueLabelKey: "value-1"}), generatePodTemplateSpec("foo", "foo-node", map[string]string{}, map[string]string{"something": "else"}), false, }, { </ins> "Same spec, the label is different, and the pod-template-hash label value is the same", generatePodTemplateSpec("foo", "foo-node", map[string]string{}, map[string]string{extensions.DefaultDeploymentUniqueLabelKey: "value-1"}), generatePodTemplateSpec("foo", "foo-node", map[string]string{}, map[string]string{extensions.DefaultDeploymentUniqueLabelKey: "value-1", "something": "else"}),

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig apps What happened: Attempting a on a deployment where the only change between revisions is the addition of a single label fails with . The addition of more than one label will successfully be detected as a Template change and the rollback will occur successfully. Example: The additional label was to the pod template (both and were tested) and a new ReplicaSet with a unique pod-template-hash was generated. revision 1 revision 2 Rollout undo Repeating the above steps but adding an additional label does successfully trigger a successful What you expected to happen: The deployment is able to be successfully rolled back. How to reproduce it (as minimally and precisely as possible): 1) Create a deployment (use example above) 2) Update the deployment by adding a single additional label to the Pod Template 3) Attempt to rollback to the revision without the single additional label Anything else we need to know?: Tested both with and Environment: Kubernetes version (use ): minikube gke Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): client: OSX High Sierra minikube: v0.25.2 / Buildroot 2017.11 gke: Container-Optimized OS from Google
thanks for reporting this! It's a bug in the template comparison function. A fix is underway.


Question: return "", err } <del> // try and get canonical path for disk and if we can't throw error vmDiskPath, err = getcanonicalVolumePath(ctx, vm.Datacenter, vmDiskPath) if err != nil { klog.Errorf("failed to get canonical path for %s on node %s: %v", vmDiskPath, convertToString(nodeName), err) return "", err </del> <ins> // try and get canonical path for disk and if we can't use provided vmDiskPath canonicalPath, pathFetchErr := getcanonicalVolumePath(ctx, vm.Datacenter, vmDiskPath) if canonicalPath != "" && pathFetchErr == nil { vmDiskPath = canonicalPath </ins> } diskUUID, err = vm.AttachDisk(ctx, vmDiskPath, &vclib.VolumeOptions{SCSIControllerType: vclib.PVSCSIControllerType, StoragePolicyName: storagePolicyName})

When we introduced we an additional check during attach disk call to check for canonical path - Now this is creating a regression in many deployments because when volume is created, canonical path gets replaced - and hence when attach is attempted k8s can't find canonical path of the volume anymore. It throws following error: There are two ways of fixing this. We could decide to use old in attach code, if we can't fetch canonical path or we could decide not to replace canonical path during provisioning. cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc:
The code that replaces canonical Path during provisioning is pretty old and hence I don't remember/know why we are doing that. But since there could be many existing PVs that now have different canonical path - I think it might be safest to merge - to prevent any regressions.


Question: if timer != nil { timer.Stop() } <ins> // Start a goroutine to drain resetChan. This is needed because we've seen // some unit tests with large numbers of goroutines get into a situation // where resetChan fills up, at least 1 call to Write() is still trying to // send to resetChan, the connection gets closed, and this case statement // attempts to grab the write lock that Write() already has, causing a // deadlock. // // See https://github.com/docker/spdystream/issues/49 for more details. go func() { for _ = range resetChan { } }() </ins> i.writeLock.Lock() close(resetChan) i.resetChan = nil i.writeLock.Unlock() break Loop } }

Appears to deadlock or otherwise hang after a few seconds and then hit the 5-minute timeout. From start of log: followed by stack traces of zillions of goroutines that mostly report being blocked for 4+ minutes. Recent instances: https://travis- https://travis- cc
I tried to reproduce the issue by run kubelet unittests 200 times (100 for v1beta1, and 100 for v1beta3), but failed to reproduce the issue. I poked around, and found the only difference between my local machine and test machine is that I am running go 1.4.2 and test machine is running 1.3 (both failure cases posted by are with 1.3.0). I remembered from online I saw a deadlock issue reported against old golang version, but haven't found that issue yet. I will continue trying to reproduce the issue.
Saw this in travis runs too: https://travis-
Yes, there have been multiple failures today. Check shippable, also.
FWIW, these are not all kubelet tests failing, but all of them have many kubelet go routines in the trace - mostly stuck in managePodLoop().
I reran a couple of thousands time last night before I leave, and all passed. Yu-Ju just pointed out that the failure posted above by is go1.4. Shippable is in a really bad shape today, tons of different tests are failed, which should be handled separately.
Ok, with some hacky debugging message into kubelet package, even I couldn't reproduce the very failure reported here, I made some progress on understanding the issue here. Basically there are two issues: 1) The flaky tests reported here are TestServeExecInContainer. All reports posted here have the same stack trace below: 2) There are tons of goroutines, especially one created for each pod. Kubelet leak those goroutines. One can see when test initial run, there are 106 goroutines. Test itself create another 41 goroutines, but none of them are created for podWorkder.
FWIW, the tests in were creating go routines it didn't need at all. I just removed them in
Thanks for cleanup. Now the number down to 91:
cc/
Did something change related to this recently?

I haven't heard any flakiness reports for a long time, and now they're starting up again. Any ideas? Sent from my iPhone
I didn't see any changes related to this feature. Actually I even couldn't reproduce it on my desktop. Please note that the comment I made above all those leftover goroutines are separate issue thought.
This has exposed a deadlock in spdystream. I'll work with to come up with a fix.
Upstream issue
Cool ncdc@ ! dchen1107 by the way, I've run across a few similar bugs recently where their reproducibility hinges mainly on how the test cluster was brought up. If the test cluster comes up right, then the tests pass 100% forever. If the cluster comes up bad, the tests fail, sometimes forever. Were you rerunning your tests against the same test cluster? Or rebuilding your test cluster between every test invocation?
This is unittests, not e2e. We saw test flakiness through travis and shippable, but never on my desktop.
This will hopefully fix the deadlock: I will do a bump commit once it's merged.
Thanks for your quick fix. I will clean up goroutines after that.
it's entirely possible there's some bugs in my exec & port forwarding code around leaking goroutines. Are you going to take an initial look to see what's sticking around?
Yes, your test leak some goroutine, but other tests too.


Question: "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/sets" <del> "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" </del> "k8s.io/kubernetes/federation/apis/federation" "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/events"

/kind bug What happened: doesn't show role Workaround: What you expected to happen: According to and How to reproduce it (as minimally and precisely as possible): I just deployed a multi master kubernetes environment with kops: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS Install tools: kops, terraform
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cli
I don't think this is a bug. The node role label is not used by kubeadm anymore. Related codes have been changed in by
Hi Thanks for the quick reply I'm actually using the as you can see in the workaround above. I'm not using kubeadm role label.
Your favorite topic came up again. Hi I do think this is a bug. As said, kubectl shouldn't show anything based on the old kubeadm master label, but it should probably on the label, but absolutely on the label. cc as you have opinions on this as well cc


Question: import ( "path/filepath" "testing" <ins> "time" </ins> "github.com/stretchr/testify/assert" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/api/v1" runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1" <ins> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" </ins> containertest "k8s.io/kubernetes/pkg/kubelet/container/testing" )

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Pod status started null, started, 01 Jan 0001 Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:38:08Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"}` Environment: Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Install tools: kubeadm 1.6 What happened: When a pod is created and starts running, shows the correct container start time in Containers -containername -State -Started. After the container is terminated, the start time changes to . In addition, shows that is . What you expected to happen: I'm expecting the start time to not change after the pod is terminated, so that afterwards I could check how long a time the container execution took. How to reproduce it (as minimally and precisely as possible): Have a running kubernetes cluster, and create a pod with the following command: kubectl create -f - <<EOF apiVersion: v1 kind: Pod metadata: name: sleep spec: containers: name: sleep image: ubuntu command: ["sleep"] args: ["30"] restartPolicy: Never EOF This created a container that sleeps for 30 seconds and quits.

During those 30 seconds, check the pod status with : ... State: Running Started: Wed, 17 May 2017 10:25:22 +0300 ... After 30 seconds, check the status of the terminated container with again: ... State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Wed, 17 May 2017 10:25:52 +0300 ... With we see that the container's start time is null: ... "state": { "terminated": { "containerID": "docker://ca0dbe0dab3bb91ad0b05ac167e44d5fbf805f736ef98f61b9e106ba4456667f", "exitCode": 0, "finishedAt": "2017-05-17T07:25:52Z", "reason": "Completed", "startedAt": null } } ... Anything else we need to know: I need to measure container run durations and this bug prevents me from doing so.
/cc PR to fix this:


Question: const ( // StatefulSetInitAnnotation if present, and set to false, indicates that a Pod's readiness should be ignored. <del> StatefulSetInitAnnotation = "pod.alpha.kubernetes.io/initialized" StatefulSetRevisionLabel  = "statefulset.beta.kubernetes.io/revision" </del> <ins> StatefulSetInitAnnotation      = "pod.alpha.kubernetes.io/initialized" ControllerRevisionHashLabelKey = "controller-revision-hash" StatefulSetRevisionLabel       = ControllerRevisionHashLabelKey </ins> ) // ScaleSpec describes the attributes of a scale subresource

Here are the hash label keys of all controllers: Deployment: DaemonSet: (new in 1.7) StatefulSet: (new in 1.7) Had a brief discussion with and about this, it should be consistent. We can use for DaemonSet and StatefulSet, and keep Deployment as-is. Deployment hash label key should be different from other two because Deployment pods hash labels is used by selector (unlike the other two controllers). We probably should have used hash annotations instead of hash labels for DaemonSet and StatefulSet, since we don't use hash labels to select their history or pods. Hash labels are chosen for consistency. This is fine since we don't modify hash label values.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Actually, controller revisions created by statefulset, daemonset has different label: // controller revision created by statefulset label: "" // controller revision created by daemonset label: controller-revision-hash: "" should be consistent?


Question: type PersistentVolumeList struct { TypeMeta `json:",inline"` <del> ListMeta `json:"metadata,omitempty"` </del> <ins> ListMeta `json:"metadata,omitempty" description:"standard list metadata; see http://docs.k8s.io/api-conventions.md#types-kinds` </ins> Items    []PersistentVolume `json:"items,omitempty" description:"list of persistent volumes"` } // PersistentVolumeClaim is a user's request for and claim to a persistent volume type PersistentVolumeClaim struct { TypeMeta   `json:",inline"` <del> ObjectMeta `json:"metadata,omitempty"` </del> <ins> ObjectMeta `json:"metadata,omitempty" description:"standard object metadata; see http://docs.k8s.io/api-conventions.md#metadata"` </ins> // Spec defines the volume requested by a pod author Spec PersistentVolumeClaimSpec `json:"spec,omitempty" description: "the desired characteristics of a volume"`

PR had a few errors in its API objects, where the "description" field errors in its JSON declarations were missing. But, the "verify-" presubmit check didn't flag them. This should be investigated and fixed.
Could you please take a look at this?


Question: return ${recreate_rc} fi <del> # Wait for instance to be recreated echo "== Waiting for instance ${instance} to be recreated. ==" >&2 while true; do new_instance_id=$(gcloud compute instances describe "${instance}"  --format='get(id)'  --project="${PROJECT}"  --zone="${ZONE}" 2>&1) && describe_rc=$? || describe_rc=$? if [[ "${describe_rc}" != 0 ]]; then echo "== FAILED to describe ${instance} ==" echo "${new_instance_id}" echo "  (Will retry.)" elif [[ "${new_instance_id}" == "${instance_id}" ]]; then echo -n . else echo "Instance ${instance} recreated." break fi sleep 1 done # Wait for k8s node object to reflect new instance id </del> <ins> # Wait for node status to reflect a new boot ID. This guarantees us # that the node status in the API is from a different boot. This # does not guarantee that the status is from the upgraded node, but # it is a best effort approximation. </ins> echo "== Waiting for new node to be added to k8s.  ==" >&2 while true; do <del> external_id=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output=jsonpath='{.spec.externalID}' 2>&1) && kubectl_rc=$? || kubectl_rc=$? </del> <ins> local new_boot_id=$("${KUBE_ROOT}/cluster/kubectl.sh" get node "${instance}" --output=jsonpath='{.status.nodeInfo.bootID}' 2>&1) && kubectl_rc=$? || kubectl_rc=$? </ins> if [[ "${kubectl_rc}" != 0 ]]; then echo "== FAILED to get node ${instance} ==" <del> echo "${external_id}" </del> <ins> echo "${boot_id}" </ins> echo "  (Will retry.)" <del> elif [[ "${external_id}" == "${new_instance_id}" ]]; then </del> <ins> elif [[ "${boot_id}" != "${new_boot_id}" ]]; then </ins> echo "Node ${instance} recreated." break <del> elif [[ "${external_id}" == "${instance_id}" ]]; then echo -n . </del> else <del> echo "Unexpected external_id '${external_id}' matches neither old ('${instance_id}') nor new ('${new_instance_id}')." echo "  (Will retry.)" </del> <ins> echo -n . </ins> fi sleep 1 done

This test is currently failing in sig-release master-upgrade suite, This could potentially become release blocking if not addressed sooner. can you please triage this failure? /kind bug /priority failing-test /priority important-soon /sig testing cc /assign
hmm, looks like cluster up etc works but some of the test case(s) are hanging and taking way too much time: ... I0417 18:59:21.467] (Will retry.)
cc ^^ assume that's somewhere inside the upgrade framework
/sig cluster-lifecycle
/assign /unassign Unlikely I will have the time or expertise to patch these, discussed out of band with and these appear to be something in the upgrade or cluster-up code, not kubetest etc..
/milestone v1.11
Debugged one of the jobs. The node upgrade is stalling out. There's logic in the node upgrade to wait for the new GCE vm to register with the k8s API. It does this by checking the node status's external ID and seeing if it matches the instance ID from the GCE vm. That logic is here: It looks like the kubelet is now registering the GCE VM's resource path and not its ID as the node's externalID. from https://k8s- Can I get input on this from ? Is this an intended change? If so, what is a good way to detect that the new node has registered with kubernetes?
cc ^^ :-)
ExternalID was apparently deprecated before kubernetes V1 API, but was never removed properly. PR was merged recently to remove the use of ExternalID Can we use the instance ID instead?
No. Instance ID is returning information like "k8s-jkns-e2e-gce-slow/us-central1-f/bootstrap-e2e-minion-group-0s1q". During a node upgrade, this ID is shared between the old and new VM. We need a way to distinguish them to make sure the new VM has come up, registered with the API server, and is healthy. Without a distinguishing ID, we could be reading the health status of the old node before we deleted it.
Ah, didn't read the issue carefully. Thanks for explaining that. Kubelet also reports machine ID/ system UUID. Are they unique in the cloud environment?

Can you point to where I can find that in the API and save me having to track it down? On Tue, Apr 17, 2018, 17:24 Yu-Ju Hong <:
or
Sorry, I should have been more specific. Where do those come from? Because I can't find that anywhere in
I think those are from . I don't think they are exposed in the gcloud. If we do use them, I think the upgrade script will have to simply check if the machine ID has changed, hmm...
Oh I see. OK I'll see if I can work with that.
It doesn't work. The machine ID is the same before and after the instance recreate. Even confirmed by looking at directly. seems to have the same value as machine ID.
Urghh...recreating an instance actually keeps the machineID. I don't have a good solution. One idea is to check if the bootID changed.
status/approved-for-milestone
/status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/reopen I still see the Upgrade test failing in the jobs below.
I still see UpgradeTest and Timeout failing in the jobs below. Are you actively investigating these or should I re-triage?
These failures seem similar to my comment in I'm would appreciate help in looping in the node team on why that is happening.
/cc can either of you please help investigate the node side of this issue to unblock This is currently on of the few tests failing our sig-release upgrade blocking suite. If this is not fixed we will have to delay beta for 1.11 which is slated for 5/15.
/cc
cc
I updated the issue at , which might explain some of the failures here.
I see the tests passing now. Thanks and for all your help.


Question: } t.Run(gvk.String(), func(t *testing.T) { <del> for i := 0; i < 50; i++ { </del> <ins> for i := 0; i < 3; i++ { </ins> item := fuzzObject(t, gvk) builder := applyconfigurations.ForKind(gvk) unstructuredRoundTripApplyConfiguration(t, item, builder)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --pull-kubernetes-unit Two new tests were just in that take significant amounts of time in CI: ~These tests also appear to be flaking regularly: /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Related: At some point we should probably adjust the timeout. I'd like to get CI to a point where it uses whatever the default is in make test.
yes.
This loops each type 50 times, I'll drop that down to say, 3 times. We honestly don't need to fuzz these that hard, they typically catch problems quick.
The failures I see in gubernator shouldn't happen now that the PR is merged. Before myPR to add this test was merged, other PRs that fields (NamespaceSelector) would cause myPR's unit tests to fail and I would need to rebase it and run . Now that it is merged, will be run by the PR author that adds a field to a type to keep code in sync.
ah, so these failures on other PRs mean they needed to regen?
Yeah, the first one adds and the fuzzer detected the apply config builder had not been regenerated by . The others have field changes too.
ok, so it's noise for PRs in progress but not flakes. looks like the runtime/timeout issue is the only flake, which you can drop iterations significantly to solve
50-3 seems like it should get us to 10-15s total for both new tests


Question: import ( "context" <ins> "math/rand" </ins> "reflect" "strings" "testing"

When updating policy labels on a namespace, existing pods are evaluated to see if they are compatible with the new policy. If there are too many pods, or if evaluation takes too long, we short circuit and warn that not all pods were checked. For that reason, we want to prioritize checking a variety of pods before checking essentially identical pods, so that unique warnings can be surfaced before reaching the pod count cutoff or evaluation timeout. Pods created by the same controller are most likely to be identical (e.g. replicas from a replicaset or job, copies of pods from a statefulset or daemonset, etc) cc
This is an interesting fix I'd like to take this up Can I take this up? I am fairly new to Kubernetes source code so might need some help
I am guessing this is where the logic you mentioned should be implemented right?
thanks for the interest; this is targeted at the beta version... right now, effort is going toward the alpha-blocking work for 1.22
Should I wait for the code freeze before working on this? Sorry for lame questions
If you want to start looking at it, that's fine, but it's possible the code in the ValidateNamespace function will change before we're ready to start merging 1.23 work, and there isn't really reviewer bandwidth to take a look at work-in-progress stuff for next release yet.
Sure, I'll just for code freeze before I start with implementation By the time I'll just assign this to myself /assign
Does it make sense to take it up now? I think we already have a couple of RC builds after the code freeze
sure, it's fine to start work in a branch in your fork and have a WIP PR open, even before master reopens for 1.23 work
Awesome, I'll start working on it Thank you
Hello! Are you still working on this? If you don't have time, I can take this issue. If not, maybe I can give some help
Since this was not an urgent issue I was working on some other issues last week I have started on this, Will be raising a PR maybe today or tomorrow I Will ask you if I get stuck Thank you


Question: // getJobFromTemplate makes a Job from a CronJob func getJobFromTemplate(sj *batchv1beta1.CronJob, scheduledTime time.Time) (*batchv1.Job, error) { <del> // TODO: consider adding the following labels: // nominal-start-time=$RFC_3339_DATE_OF_INTENDED_START -- for user convenience // scheduled-job-name=$SJ_NAME -- for user convenience </del> labels := copyLabels(&sj.Spec.JobTemplate) annotations := copyAnnotations(&sj.Spec.JobTemplate) // We want job names for a given nominal start time to have a deterministic name to avoid the same job being created twice

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "scheduledjob" (and try to understand from the title whether this could be related to my issue) "find job" Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request. Kubernetes version (use ): ~ Environment: Cloud provider or hardware configuration: AWS via kube-aws 0.93-ish What happened: I'm using scheduled jobs, and then try to review the logs of the job executions. I have multiple scheduled jobs in the system. Generally this works via: ... search for the ones that seem to relate to my scheduled job name . Repeat this if there are more than one execution. the whole list for previous jobs What you expected to happen: I'd love to be able to filter the initial job list already to only include jobs created by a specific scheduled job, something like . For that to happen I need scheduledjob to label the created job, just like a job already labels the pods executing it.
can you include labels in the cronjob's jobTemplate spec?
Yes, that is certainly possible. I filed this ticket mainly because I think this should happen automatically: Any resource produced by a resource I created should be labeled such that I can find these resources back easily.
/sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
From my perspective this is still something I'd like to see.
I've given this a go. There was actually a todo for it in the comments.
/remove-lifecycle rotten
Lovely, thanks a lot!


Question: <del> # Copyright (C) 2017 SUSE LLC. All rights reserved. # Use of this source code is governed by a BSD-style # license that can be found in the LICENSE file. language: go go: - 1.13.x - 1.16.x - tip arch: - AMD64 - ppc64le os: - linux - osx script: - go test -cover -v ./... notifications: email: false </del>

One of the dependencies of k8s, filepath-securejoin, has a following vulnerability discovered: It has been patched in version 0.2.4, however k8s is still using version 0.2.3. Potential fix: updating the dependency. A small PR will follow. /sig security
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig security
/kind bug /sig security


Question: hostConfig.SecurityOpt = append(hostConfig.SecurityOpt, "no-new-privileges") } <del> hostConfig.MaskedPaths = sc.MaskedPaths hostConfig.ReadonlyPaths = sc.ReadonlyPaths </del> <ins> if !hostConfig.Privileged { hostConfig.MaskedPaths = sc.MaskedPaths hostConfig.ReadonlyPaths = sc.ReadonlyPaths } </ins> return nil }

Is this a BUG REPORT or FEATURE REQUEST?: kind bug What happened: After execute kubeadm init during kube-proxy pod initialization there is CrashLoopBackOff error all the time Logs from pod: [centos ~]$ kubectl logs kube-proxy-8gs45 -n kube-system What you expected to happen: kube-proxy pod run properly How to reproduce it (as minimally and precisely as possible): follow instruction: try to execute: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools:Others:
/sig network
I have exactly the same problem with the 1.12.0-beta.1. Does kube-proxy really need access to /proc/sys/* in this case?
Same problem with Ubuntu
Does that mean kube-proxy can only be run outside of container?
/sig cluster-lifecycle
/milestone v1.12
Looks like docker inspect shows masked and read-only paths (looking at ) even when the pod is
It being close to release, since this is not urgent, please save this for 1.13! /milestone clear /milestone v1.13
/milestone v1.13
/priority critical-urgent /milestone v1.12 Sorry the activity and urgency is reflected in the PR :)
welp! sorry about that!
I'm still seeing this on RC1, did this land before the release was cut? /cc
update from :
I just verified on release-1.12 branch and it's good.


Question: "net.ipv4.ip_unprivileged_port_start", "net.ipv4.ip_local_reserved_ports", "net.ipv4.tcp_keepalive_time", <ins> "net.ipv4.tcp_fin_timeout", "net.ipv4.tcp_keepalive_intvl", "net.ipv4.tcp_keepalive_probes", </ins> }, }, }

Add the following to be safe sysctls: net.ipv4.tcpfintimeout net.ipv4.tcpkeepaliveintvl net.ipv4.tcpkeepaliveprobes These are related settings to net.ipv4.tcpkeepalivetime that was enabled in These are related ipv4 TCP settings for tuning network connections between a client and a pod.
/sig network
May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node.
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : May you help provide some tests to ensure that setting this sysctl can't affect any other pods or the node. /sig nodes Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
I adjusted it with the plugin.
Would this change apply at the node level or the pod level? I assumed it would be at the pod level, and should not affect any other pods--however Kubernetes does not natively have any limits on network connections that I am aware of--but only the pod with the changed net settings would be affected.
To verify that a sysctl has an effect on other pods, we need to verify that it's a namespace sysctl.
Here's a simple test Use net.ipv4.tcpfintimeout sysctl to create a container check if it set on the host =It is the default value on host run another container attach to the network namespace of the one where the sysctl is set =results in sysctl being set Verify in another container under a different network namespace that it is not set =It is the default value According to the test results, net.ipv4.tcpfintimeout is a namespaced sysctl
I likewise verified and , which are also namespaced sysctl, and I believe we can add them to the safe sysctl /assign
Probably need to wait for to finish before start working on it
/triage accepted


Question: } // podFitsOnNode checks whether a node given by NodeInfo satisfies the given predicate functions. <ins> // For given pod, podFitsOnNode will check if any equivalent pod exists and try to reuse its cached // predicate results as possible. </ins> // This function is called from two different places: Schedule and Preempt. // When it is called from Schedule, we want to test whether the pod is schedulable // on the node with all the existing pods on the node plus higher and equal priority

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Today, . This seems unnecessary and should be done once for all the nodes. Is this a BUG REPORT or FEATURE REQUEST?: /kind enhancement /sig scheduling cc:
/assign Thanks, will send fix per slack discussion.
/cc


Question: "cors.go", "doc.go", "goaway.go", <ins> "hsts.go", </ins> "longrunning.go", "maxinflight.go", "priority-and-fairness.go",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add the "Strict-Transport-Security" header to the API server responses. Why is this needed: Nessus scanner treats this as a Medium Risk in kubernetes 1.18.10.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig auth


Question: fs.StringVar(&s.OIDCGroupsClaim, "oidc-groups-claim", "", ""+ "If provided, the name of a custom OpenID Connect claim for specifying user groups. "+ <del> "The claim value is expected to be an array of strings. This flag is experimental, "+ </del> <ins> "The claim value is expected to be a string or array of strings. This flag is experimental, "+ </ins> "please see the authentication documentation for further details.") fs.Var(&s.RuntimeConfig, "runtime-config", ""+

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No, already discussed on sig-auth What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): confirmed issue on sig-auth channel Is this a BUG REPORT or FEATURE REQUEST? (choose one): Depends on interpretation <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-08-02T00:54:53Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: CoreOS Vagrant Image OS (e.g. from /etc/os-release): NAME=CoreOS ID=coreos VERSION=1122.0.0 VERSIONID=1122.0.0 BUILDID=2016-07-27-0739 PRETTYNAME="CoreOS 1122.0.0 (MoreOS)" ANSICOLOR="1;32" HOMEURL="" BUGREPORTURL="" Kernel (e.g.

): Linux localhost 4.7.0-coreos SMP Wed Jul 27 07:30:04 UTC 2016 x8664 Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz GenuineIntel GNU/Linux Install tools: Vagrant Others: What happened: The claim named in --oidc-groups-claim MUST be list of strings, even if there is only one value. Talking to this is by design since its assumed that users will have more then one group. What you expected to happen: The OIDC authenticator should accept either a list of strings or a single string since: There is no concept of schema that marks an attribute as single value or multi value in JSON or a JWT There is no standard for representing groups in a JWT There is no standard for representing groups in OIDC An OIDC should be flexible in how it accepts data. How to reproduce it (as minimally and precisely as possible): Deploy with RBAC enabled Create a role binding for the group "admin" Generate a JWT such as the one that does NOT work below User won't be able to login Generate a JWT where the role is multi-value User will be authorized Does NOT Work: DOES Work: Anything else do we need to know:
These should be called groups, not roles, right? Those are different. And, is it normal for a JWT to contain keys which are arrays? That seems weird.
where do you see role defined? you mean the user_role claim?
Claims can be structured, yes. The name of the groups claim is arbitrary. Could easily have been "group". In my mind, this is all about enabling mapping a "groupish" claim from whatever IDP issued the JWT, even if it is a single value instead of an array.
That's the way it works currently. Yeah I'm fine with this.
Okay.


Question: timeout: 90 emails: '$DEFAULT_RECIPIENTS, beeps@google.com' test-owner: 'beeps' <ins> - 'gce-es-logging': description: 'Run [Feature:Elasticsearch] tests on GCE using the latest successful build.' timeout: 90 emails: '$DEFAULT_RECIPIENTS, mixia@google.com' test-owner: 'mixia' </ins> jobs: - 'kubernetes-e2e-{suffix}'

It failed in gce suite. I honestly have no idea under which team it falls:
Another failure: bumping to p0. - can you find an owner, or at least an appropriate team for this?
owns logging (see team description).
Cannot reproduce failure so far. Leaving it to run overnight...
Still cannot reproduce the exact failure... Increase verbosity...
There were a lot of such failures over the weekend:
Moving this test to flaky (), as it's only 66% stable and is blocking the build quite often: please prioritize this.
This test was passing 100% of the time until Build (Dec 5, 2015 4:13:38 PM). After that, it failed 50% of the time (10 failures in 20 runs). Clearly a PR that merged shortly before the above time introduced the flakyness. That PR should be rolled back.
hmm - I looked into PRs that were merged around that time, and actually nothing looks suspicious to me
This test is 100% successful in flaky test suite so far. I suspect some other e2e test is impacting this one. Adding more sanity checks... (root)/Kubernetes%%20suite/ClusterlevelloggingusingElasticsearchshouldcheckthatlogsfrompodsonallnodesareingestedinto_Elasticsearch/
Let it run in flaky suite overnight. If everything goes well, will unmark it.
I can't follow any of the links but if I could get a bit more information about what is going wrong I could look into the Elasticsearch test if you think that is the issue rather than something else having a bad impact on it.
Thanks for offering to help. Here are the failure symptoms in Jenkins: cluster in red status. I bump into this once with my test cluster. It is cause by es split brain problem. Fix already merged. all log lines from one node. I cannot reproduce this. My guess is the fluentd pod on a node is not working properly. So I validation for fluentd pods. Can you think of anything else?
Can you please point me to the fix for the split brain problem? Is that ? So that works by making sure there are always two master nodes? Is 2 really a good idea for quorum? As suggested here a choice of 2 might not be desirable?

Missing log lines indeed mean a synthetic logging pod did not start for some reason -- without more log information it's hard to work out exactly why this occurred. If I get a moment I will try to do a repo. Thanks.
minimummasternodes set to 2 does not mean having 2 master nodes. It means the es cluster needs to have at least 2 nodes before electing a master. If network partition happened to a es non-master node, it will wait until it could reach out to at least one other node to start electing master. Hence it will prevent the split brain problem. Based on the log in jenkins, all synthetic logging pod ran successfully.
It gets flaky again... I am able to reproduce the problem by running DaemonRestart and Elasticsearch test together. It looks like the DaemonRestart test somehow disrupts the ES cluster. ES cluster gets into bad state, hence failing the es test. Confirmed with This particular test restarts all the containers.
It's a recent regression. Found the cause and will upload a fix soon.
- please monitor this test and close this bug if there won't be any failures in some time.
Test is green for the last 40 builds. Closing the issue.
This test is still marked as at . please send a PR to promote it out of ?
Since is the owner of the issue. I'll let him decide whether the test is ready.
The test has been successful lately. Want to add one more check to improve resilience. I had a pending PR to remove label. Will try to merge it on Monday if everything goes well during the weekend.
I think that is because elasticsearch query return unexpect results. should solve it.
Weirdly hit the same unrelated pr twice, but it looks like the fix just got merged
Still flaking:
what's next on this? Seems to not be resolved...
Moved back into flaky in , is trying for repro/solidifying the test IIUC
Trying to reproduce the problem. It is running successful in flaky suite and my local cluster so far... Will try to run it with other tests together.
Two types of failure: cluster in bad state. This only happens when running with other disruptive tests. Never failed in suite where it is not running with disruptive tests. ES resilience can be improved by log lines in Elasticsearch.

I am able to reproduce this once. It is due to fluentd is hanged. Filed a issue to upstream. Request to mark this test as [Feature] since it depends on two 3rd party components.
The risk here is that ES is a default cluster addon. IMO it's fine not testing it for critical builds, but the risk is we'll ship something that straight crashloops. So if we move it to feature I'd ask that you add a smaller test that verifies basic functionality to a blocking suite (ping es, check that it accepts some logs). Or make it non-default.
Good point. I will work on first. It has to be done anyway. It will also make future ES updates easier. BTW, ES addon is not default for GCE/GKE.
Really? my e2e cluster always has it, why are we testing with it then?
For GCE:
Yeah - another place where test and 'production' configs differ, but I guess we need to test elasticsearch for 'fully opensource' version, thus it's justified...
That's [Feature] then, just like we test flannel. Not something to bring up as an addon on every e2e clusters.
We probably need some suite with all 'important' options (e.g. Features) turned on, on top of dedicated suite for every feature. Ugh...


Question: import ( "bytes" "encoding/json" <ins> "fmt" </ins> "io/ioutil" "reflect" "time"

We have , but there isn't a label in the k/k repo. There is only . Do we need to add a label or rename to ?
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/sig ui :smiley:
/sig contributor-experience
/cc
I just it. Fixed?
Yeah. BTW, the page has a bad md format. It also should be fixed.
Feel free to send a PR to fix :)
As Piotr is the owner, I thought let him fix:) If it's Okay, I will do. Thanks Christoph!
anyone can contribute to the kubernetes/community repo too :) the PR would be reviewed like any other (usually the sig-lead for that sig would review)
Okay:)
Yep just fix things when you can. That's just faster :)
FWIW this lead me to open

I can see that this message is flooding my minion logs so I was curious what should this config file looks like. I was trying to find some example, but with no luck. Also is this config file created during the runtime? What actually creates it? Shouldn't we just give up when the config file does not exists? Also there are better ways (maybe) to do polling if the file exists or not. I meaning you can use inotify () instead of an infinite loop + sleep. I'm happy to make a patch if you think it is a good idea. PS: I know that you put the timeout 5 seconds before retries.
Thanks for the report. I think this is just a bug-- we should check whether it exists and not pollute the log. I'm pretty sure we don't currently even use that file. Also we should NOT be reading the proxy's config from a file in /tmp where anybody can write it. Bad default location.
Proxy config sounds like an /etc thing, same for kubelet's container manifests from disk
Are we interested in polling for the configuration files changes? Or do we want to read the settings just once, as an ordinary unix binary?
Yes, polling is a requirement. This is a live config file like the kubelet's machine files.


Question: Long:    editLong, Example: fmt.Sprintf(editExample), Run: func(cmd *cobra.Command, args []string) { <del> args = append([]string{"configmap"}, args...) </del> err := RunEdit(f, out, errOut, cmd, args, options) cmdutil.CheckErr(err) }, ValidArgs:  validArgs, ArgAliases: argAliases, } <del> addEditFlags(cmd, options) cmd.AddCommand(NewCmdEditConfigMap(f, out, errOut)) return cmd } func addEditFlags(cmd *cobra.Command, options *resource.FilenameOptions) { </del> usage := "to use to edit the resource" cmdutil.AddFilenameOptionFlags(cmd, options, usage) cmdutil.AddValidateFlags(cmd)

kubectl edit is not working on HEAD. It's working on 1.5.


Question: allErrs.CELErrors = append(allErrs.CELErrors, field.Invalid(fldPath.Child("x-kubernetes-validations").Index(i).Child("messageExpression"), schema.XValidations[i], cr.MessageExpressionError.Detail)) } else { if cr.MessageExpression != nil { <del> if cr.MessageExpressionMaxCost > StaticEstimatedCostLimit { </del> <ins> if !opts.suppressPerExpressionCost && cr.MessageExpressionMaxCost > StaticEstimatedCostLimit { </ins> costErrorMsg := getCostErrorMessage("estimated messageExpression cost", cr.MessageExpressionMaxCost, StaticEstimatedCostLimit) allErrs.CELErrors = append(allErrs.CELErrors, field.Forbidden(fldPath.Child("x-kubernetes-validations").Index(i).Child("messageExpression"), costErrorMsg)) }

We were using cel-go v0.12.7 in Kubernetes 1.27 release and cel-go v0.16.0 in Kubernetes 1.28 release. There are many changes from cel-go side between those two releases and some may break backward compatible of CRD validation rules. An example would be the estimated cost for CRD validation rules(Thanks to who brought this to our attention with the following example). In the example CRD validation rules below: It will pass perfect in Kubernetes 1.27 version but fail in Kubernetes 1.28 with error: This is due to the estimator fix happened in cel-go(, ) The changes should not cause backward incompatible of CRD validation rules. OR a stability test suite should be in place to prevent the changes causing backward compatible issue before GA. Apply the above CRD in Kubernetes 1.27 and 1.28, 1.27 cluster should pass and 1.28 cluster should fail. Since the fix in cel-go side makes sense to me and the feature is in Beta now, I am not quite sure how strict we should be on backward compatible. cc As a followup, a stability test suite for estimate cost is worth adding before GA to make sure any future changes be caught before the feature promoted to GA. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi I agree that the changes in cel-go should not break the backward compatibility of CRD validation rules, especially for users who have already defined and deployed their custom resources using the previous version of Kubernetes. One suggestion I have is to use a version flag for the CEL library, so that users can specify which version of cel-go they want to use for their CRD validation rules. This way, they can avoid unexpected errors or behavior changes when upgrading their Kubernetes cluster.

For example, they can add something like in their CRD schema to indicate that they want to use the old version of cel-go.
Another suggestion I have is to use a cost limit parameter for the CEL library, so that users can adjust the budget for their CRD validation rules. This way, they can fine-tune the trade-off between performance and complexity of their rules. For example, they can add something like in their CRD schema to indicate that they want to allow more expensive rules.
I agree this is the type of bug fix that is legitimate to make during beta. when the size changes came in, did our existing size estimation test coverage notice this cost increased? If not, is there a systemic change we could make to exercise each type of estimation combination so we have visibility to any future changes. Existing persisted CRD objects should continue to work as-is on upgrade, but the CEL rules and/or schema would be required to fit within fixed cost estimates before making future updates/changes to the CEL rules there should be unit tests ensuring updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates, but can you verify that is the case? That's not workable, both from an implementation perspective and from a fragmentation perspective. We explicitly don't want to allow opting into loopholes / bugs in cost estimation that allow CRD authors to make API servers do excessive amounts of work. Similar to above, the responsibility for API server resource consumption lies with the kube-apiserver admin, not the CRD author, so exposing this in the API is not something I would expect.
/assign
Thank you! I guess the next step would be: the existing test suite regarding with the cel cost evaluation. Add estimated cost tests for every single operation, function, type, etc if it's not existing now. updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates. Currently it is not working like this but would definitely help here. cc , please correct me if I'm wrong :) Those should be done before the feature goes to GA. I am not sure about the policy of a GA feature relying on alpha/beta feature is acceptable or not. is going to Beta in this release. The worst case is we have to wait further until the feature is ready.

I don't think that helps... CRD ratcheting is about validation of the custom resources. This is about validation of the CRD itself.
thanks for clarifying! Are you suggesting while updating CRD, if the rules matching what's in the existing object, we should skip cost estimation check? That makes sense. I could start a PR if my understanding aligns :)
something along those lines, though now that I think about it, other changes in the schema could affect the number of times a CEL expression could be run (dropping or increasing maxItems, maxProperties, etc) we already collect preexisting expressions when updating a CRD in order to allow already-persisted expressions (possibly persisted via a newer API server) making use of new CEL features: I think that is what I was thinking of. I would probably focus on: coverage of the edges of the cost estimation so we don't pick up changes there without knowing coarse-grained skipping of schema/cel validation for a particular version in a CRD for updates which don't change anything under that version's schema (would let someone make unrelated updates to the CRD and leave an existing version's schema / CEL rules alone)
Thank you! We have those work planned for the current cycle. /assign /assign