Question: wantStatus:       framework.NewStatus(framework.Unschedulable, ErrReasonMaxVolumeCountExceeded), }, { <ins> newPod:           inTreeInlineVolPod, existingPods:     []*v1.Pod{inTreeTwoVolPod}, filterName:       "csi", maxVols:          2, driverNames:      []string{csilibplugins.AWSEBSInTreePluginName, ebsCSIDriverName}, migrationEnabled: true, limitSource:      "node", test:             "nil csi node", }, { </ins> newPod:           pendingVolumePod, existingPods:     []*v1.Pod{inTreeTwoVolPod}, filterName:       "csi",

handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.


Question: function kube::release::package_salt_tarball() { echo "+++ Building tarball: salt" <del> local release_stage="${LOCAL_OUTPUT_ROOT}/release-stage/salt/kubernetes" </del> <ins> local release_stage="${RELEASE_STAGE}/salt/kubernetes" </ins> rm -rf "${release_stage}" mkdir -p "${release_stage}"

Sometimes it is nice to have non-tar'd binaries for releases. We should upload those along with the our packaged tarballs. This will help fix


Question: "//staging/src/k8s.io/apimachinery/pkg/api/resource:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/errors:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", ],

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: A failed Prioritize Map Function caused Scheduler break down. What you expected to happen: Pod remains Pending and describe pod will show errors of failed functions. How to reproduce it (as minimally and precisely as possible): It happens when some prioritize's map functions return with error. Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
As shown above, if priorityConfigs[i].Map(pod, meta, nodeInfo) returns with error, there will be no chance for results[i + 1] to initialize its value, thus causing panic when calling config.Reduce(pod, meta, nodeNameToInfo, results[i+1])
/sig scheduling
I opened a PR to initialize results with real host name instead of empty string.


Question: checkpoint, err := ds.checkpointHandler.GetCheckpoint(podSandboxID) // Return empty portMappings if checkpoint is not found if err != nil { <ins> if err == utilstore.ErrKeyNotFound { return nil, nil } </ins> return nil, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug CheckpointNotFoundError has been completely removed by But GetPodPortMappings method was not updated to return nil, when a checkpoint was not available. What happened: The CNI network (that is done for the shared containers only for windows) call would fail in this case, which is breaking CNI in windows. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
Related issue
/sig node /sig windows
/assign
Previous PR for tracking


Question: ], embed = [":go_default_library"], deps = [ <ins> "//pkg/kubelet/config:go_default_library", </ins> "//pkg/volume:go_default_library", "//pkg/volume/testing:go_default_library", "//staging/src/k8s.io/api/core/v1:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Deletion of a Pod with an ISCSI volume in mode results in an error during , leaving an ISCSI target logged into on the kubelet node despite the target being unused. In my case, the ISCSI Target server enforces a single session per LUN, which blocks the Pod from starting on another Node in the cluster. What you expected to happen: For the kubelet to issue a logout request for the given target once the refcounter hits zero. How to reproduce it (as minimally and precisely as possible): Have an ISCSI Target server Create a target/LUN that contains an ext4 filesystem. Have a Node configured as an ISCSI Initiator capable of logging into the those targets. Create a Pod on the Node that includes an volume referring the LUN, using (rather than ) mode. minimally: Delete the Pod Wait for the Pod to be destroyed and the related ISCSI block device to be removed. on the Node will show the target is still logged in Kubelet error: Anything else we need to know?: Environment: Kubernetes version (use ): 1.16.2 Cloud provider or hardware configuration: Reproducible on a bare metal (Intel NUC) lab and DigitalOcean DOKS. OS (e.g: ): Ubuntu 18.04 Kernel (e.g. ): 4.15 (NUC) / 4.19 (DigitalOcean) Install tools: Ubuntu package /sig storage


Question: // at least "minFeasibleNodesToFind" feasible nodes no matter what the value of this flag is. // Example: if the cluster size is 500 nodes and the value of this flag is 30, // then scheduler stops finding further feasible nodes once it finds 150 feasible ones. <del> // When the value is 0, default percentage (50%) of the nodes will be scored. </del> <ins> // When the value is 0, default percentage (5%--50% based on the size of the cluster) of the // nodes will be scored. </ins> PercentageOfNodesToScore int32 `json:"percentageOfNodesToScore"` // DEPRECATED.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Kubernetes scheduler has that specifies what percentage of the cluster nodes should be scored in each scheduling cycle. The scheduler stops scanning more nodes once that many nodes are found feasible for running the pod under consideration. The default value of this config option is 50 (for 50%). We should let the scheduler determine this value based on the size of the cluster when this config option is not provided. I'd propose this formula for finding the percentage: . This is a linear formula that gives us 50 when the cluster size is 100 nodes and 10 for a 5000 node cluster. The scheduler already has a lower bound of 50 nodes per scheduling cycle. As a part of this change, we should also add a 5% lower bound for the formula (If the result of the formula is below 5, we should return 5). Why is this needed: To improve scheduler performance in large clusters. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature /sig scheduling
/assign /remove-help
/assign wgliang
Thanks


Question: podRegistry     PodRegistry } <ins> func findPort(manifest *api.ContainerManifest, portName util.IntOrString) (int, error) { if ((portName.Kind == util.IntstrString && len(portName.StrVal) == 0) || (portName.Kind == util.IntstrInt && portName.IntVal == 0)) && len(manifest.Containers[0].Ports) > 0 { return manifest.Containers[0].Ports[0].ContainerPort, nil } if portName.Kind == util.IntstrInt { return portName.IntVal, nil } name := portName.StrVal for _, container := range manifest.Containers { for _, port := range container.Ports { if port.Name == name { return port.ContainerPort, nil } } } return -1, fmt.Errorf("no suitable port for manifest: %s", manifest.ID) } </ins> func (e *EndpointController) SyncServiceEndpoints() error { services, err := e.serviceRegistry.ListServices() if err != nil {

What would you like to be : We'd like to have the Prometheus stack () running in our scalability tests. Why is this needed: There are couple of reasons: will simplify writing SLI assertions/tests that are based on metrics not necessarily exported on master. Some examples here are the new In-Cluster Network Programming Latency SLI (exported in kube-proxy) or soon to be implemented RTT SLI where the metric will be exported in regular pods (probes). With Prometheus stack, collecting and aggregating such metrics is very simple. -prometheus provides many useful metrics about the state of the cluster, such metrics might be invaluable when debugging scalability test failures. mimics what customers do in their cluster, making the test setup more realistic.
/sig scalability
/assign
The GCE 5K Node Performance test with enabled Prometheus completed successfully. We can enable Prometheus in all tests.
Looks like kubemark-500 has some issue, test fails often with Kubemark-5000 also failed once with Will disable Prometheus in kubemark, will work on fixing it when I'm back from my vacation.
For the first kind of the failure that kubemark-500 suffered from, i.e. the root cause is Looks like the token for the "prometheus-operator" service account was not ready, as we can see in the logs it was created after the POST to create the servicemonitor was called.
It looks like the second failure (kubemark-5000) was caused by the prometheus-k8s pods not scheduled by the kube-scheduler. This is strange but may be related to other issues we had. Hard to investigate what really happened there, will leave it for now. Will re-enable prometheus tomorrow and will see if it happens again.
Closing, as this is basically done in a minimal version. Other improvements will be tracked in separate issues. /close
Closing this issue. <detailsIn response to : Closing, as this is basically done in a minimal version. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: repeated Toleration tolerations = 22; // HostAliases is an optional list of hosts and IPs that will be injected into the pod's hosts <del> // file if specified. This is only valid for non-hostNetwork pods. </del> <ins> // file if specified. </ins> // +optional // +patchMergeKey=ip // +patchStrategy=merge

there is a note above hostAliases that this feature does not work with hostNework=true, yet it seems it does: when testing with a hostNetwork pod + hostAliases, there are no errors in kubelet logs at --v=10 related to this, in fact i don't see any hostAlias entries at all in there. i could not find anything blocking this in the kubelet, in fact there is normal handling of hostNetwork: i could not find any validation performed on the hostAliases field of PodSpec if hostNetwork is true on the API level. i could not find any unit tests or e2e tests related to this. the godoc should contain valid information if the feature is not supposed to work, it must be patched to not have the entries in the hosts file (perhaps?) we should have tests that ensure this type of requirement is guaranteed these steps confirm that a static pod with hostNework (deployed by kubeadm) does make use of hostAliases. create a kubeadm single node cluster directly with or one with modify the to include:wait some time for the new static pod spec to resync find the PID of the kube-apiserverget the hosts file n/a latest master 1.30.0-alpha.0 not applicable <details</detailskubeadm containerd v1.6.14 kindnet add test: update godoc:
/sig node network cluster-lifecycle /kind documentation
as reported by the hostAliases sometimes are not mounted for hostNetwork static pods, which is concerning: related slack thread:
At minimum the comment is wrong I need to be updated and we need e2e tests :/
found this, support was a while back: i will PR the API docs.
We identified an edge case here with Kubernetes v1.29. If a cloud provider did not yet initialise the node's , the code for creating the file including the hostAliases does not get run. To be concrete, in: is empty in this case which leads to to be . I was able to trace it back to this change which leads to this behaviour for static pods: For confirmation: I reverted the above commit, rebuilt kubelet and it got back to the old behaviour.

there is no perfect solution, but reverting is wrong because opens another bug, we need to study this more carefully but in the meantime installers that know the nodes addresses beforehand should set the --node-ip flag or not use cloud-provider external
yeah, as i mentioned downstream, revering is out of the question. it fixes some long standing problems. we might have to document somewhere this limitation in a cloud provider env.
i think it might be possible for us to special case this so that the hostAlias management does not wait for the node IP in a CP env, under certain conditions. just a speculation.
PRs add test: update godoc:
/triage accepted looks like it is being worked on already


Question: err: nil, }, { <ins> name:    "existing labels from dynamic provisioning are not changed", handler: newPersistentVolumeLabel(), pvlabeler: mockVolumeLabels(map[string]string{ v1.LabelZoneFailureDomain: "domain1", v1.LabelZoneRegion:        "region1", }), preAdmissionPV: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "awsebs", Namespace: "myns", Labels: map[string]string{ v1.LabelZoneFailureDomain: "existingDomain", v1.LabelZoneRegion:        "existingRegion", }, Annotations: map[string]string{ persistentvolume.AnnDynamicallyProvisioned: "kubernetes.io/aws-ebs", }, }, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ AWSElasticBlockStore: &api.AWSElasticBlockStoreVolumeSource{ VolumeID: "123", }, }, }, }, postAdmissionPV: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name:      "awsebs", Namespace: "myns", Labels: map[string]string{ v1.LabelZoneFailureDomain: "existingDomain", v1.LabelZoneRegion:        "existingRegion", }, Annotations: map[string]string{ persistentvolume.AnnDynamicallyProvisioned: "kubernetes.io/aws-ebs", }, }, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ AWSElasticBlockStore: &api.AWSElasticBlockStoreVolumeSource{ VolumeID: "123", }, }, NodeAffinity: &api.VolumeNodeAffinity{ Required: &api.NodeSelector{ NodeSelectorTerms: []api.NodeSelectorTerm{ { MatchExpressions: []api.NodeSelectorRequirement{ { Key:      v1.LabelZoneRegion, Operator: api.NodeSelectorOpIn, Values:   []string{"existingRegion"}, }, { Key:      v1.LabelZoneFailureDomain, Operator: api.NodeSelectorOpIn, Values:   []string{"existingDomain"}, }, }, }, }, }, }, }, }, err: nil, }, { name:    "existing labels from user are changed", handler: newPersistentVolumeLabel(), pvlabeler: mockVolumeLabels(map[string]string{ v1.LabelZoneFailureDomain: "domain1", v1.LabelZoneRegion:        "region1", }), preAdmissionPV: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name: "awsebs", Namespace: "myns", Labels: map[string]string{ v1.LabelZoneFailureDomain: "existingDomain", v1.LabelZoneRegion:        "existingRegion", }, }, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ AWSElasticBlockStore: &api.AWSElasticBlockStoreVolumeSource{ VolumeID: "123", }, }, }, }, postAdmissionPV: &api.PersistentVolume{ ObjectMeta: metav1.ObjectMeta{ Name:      "awsebs", Namespace: "myns", Labels: map[string]string{ v1.LabelZoneFailureDomain: "domain1", v1.LabelZoneRegion:        "region1", }, }, Spec: api.PersistentVolumeSpec{ PersistentVolumeSource: api.PersistentVolumeSource{ AWSElasticBlockStore: &api.AWSElasticBlockStoreVolumeSource{ VolumeID: "123", }, }, NodeAffinity: &api.VolumeNodeAffinity{ Required: &api.NodeSelector{ NodeSelectorTerms: []api.NodeSelectorTerm{ { MatchExpressions: []api.NodeSelectorRequirement{ { Key:      v1.LabelZoneRegion, Operator: api.NodeSelectorOpIn, Values:   []string{"region1"}, }, { Key:      v1.LabelZoneFailureDomain, Operator: api.NodeSelectorOpIn, Values:   []string{"domain1"}, }, }, }, }, }, }, }, }, err: nil, }, { </ins> name:    "GCE PD PV labeled correctly", handler: newPersistentVolumeLabel(), pvlabeler: mockVolumeLabels(map[string]string{

API server latency for PV creation depends on underlying cloud API latency. We have admission plugin that examines incoming PV and adds topology labels and node selector: When the cluster (and cloud) are under pressure, e.g. lot of PVCs are created and dynamically provisioned at one time and attached to nodes, API server latency increases. We've seen cases when we ran out of cloud API request quota and the admission plugin needed to sleep for couple of seconds (on AWS). There should be an easy fix: check, if the created PV already has all necessary labels and topology information and ask the cloud for them only when they're missing. Dynamically provisioned volumes already have these labels. The admission plugin will then ask the cloud only for manually provisioned PVs, which we believe is just a fraction of all PVs in the cluster. There is one glitch: previously, when user created a PV with wrong topology labels, the admission plugin corrected them. With the fix suggested above, this PV won't be corrected. /sig storage cc
Implementation of the above:


Question: <ins> package client type Series struct { Name    string          `json:"name"` Columns []string        `json:"columns"` Points  [][]interface{} `json:"points"` } func (self *Series) GetName() string { return self.Name } func (self *Series) GetColumns() []string { return self.Columns } func (self *Series) GetPoints() [][]interface{} { return self.Points } </ins>

Due to changes in field selectors (see ) heapster was almost certainly busted, but continues to pass. This seems incorrect.


Question: err = waitForResourceQuota(f.ClientSet, f.Namespace.Name, resourceQuotaNotBestEffort.Name, usedResources) Expect(err).NotTo(HaveOccurred()) }) <ins> It("Should be able to update and delete ResourceQuota.", func() { client := f.ClientSet ns := f.Namespace.Name By("Creating a ResourceQuota") quotaName := "test-quota" resourceQuota := &v1.ResourceQuota{ Spec: v1.ResourceQuotaSpec{ Hard: v1.ResourceList{}, }, } resourceQuota.ObjectMeta.Name = quotaName resourceQuota.Spec.Hard[v1.ResourceCPU] = resource.MustParse("1") resourceQuota.Spec.Hard[v1.ResourceMemory] = resource.MustParse("500Mi") _, err := createResourceQuota(client, ns, resourceQuota) Expect(err).NotTo(HaveOccurred()) By("Getting a ResourceQuota") resourceQuotaResult, err := client.CoreV1().ResourceQuotas(ns).Get(quotaName, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) Expect(resourceQuotaResult.Spec.Hard[v1.ResourceCPU]).To(Equal(resource.MustParse("1"))) Expect(resourceQuotaResult.Spec.Hard[v1.ResourceMemory]).To(Equal(resource.MustParse("500Mi"))) By("Updating a ResourceQuota") resourceQuota.Spec.Hard[v1.ResourceCPU] = resource.MustParse("2") resourceQuota.Spec.Hard[v1.ResourceMemory] = resource.MustParse("1Gi") resourceQuotaResult, err = client.CoreV1().ResourceQuotas(ns).Update(resourceQuota) Expect(err).NotTo(HaveOccurred()) Expect(resourceQuotaResult.Spec.Hard[v1.ResourceCPU]).To(Equal(resource.MustParse("2"))) Expect(resourceQuotaResult.Spec.Hard[v1.ResourceMemory]).To(Equal(resource.MustParse("1Gi"))) By("Verifying a ResourceQuota was modified") resourceQuotaResult, err = client.CoreV1().ResourceQuotas(ns).Get(quotaName, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) Expect(resourceQuotaResult.Spec.Hard[v1.ResourceCPU]).To(Equal(resource.MustParse("2"))) Expect(resourceQuotaResult.Spec.Hard[v1.ResourceMemory]).To(Equal(resource.MustParse("1Gi"))) By("Deleting a ResourceQuota") err = deleteResourceQuota(client, ns, quotaName) Expect(err).NotTo(HaveOccurred()) By("Verifying the deleted ResourceQuota") _, err = client.CoreV1().ResourceQuotas(ns).Get(quotaName, metav1.GetOptions{}) Expect(errors.IsNotFound(err)).To(Equal(true)) }) </ins> }) var _ = SIGDescribe("ResourceQuota [Feature:ScopeSelectors]", func() {

What would you like to be : e2e test suites for: PUT /api/v1/namespaces/{namespace}/resourcequotas/{name} DELETE /api/v1/namespaces/{namespace}/resourcequotas/{name} Why is this needed: The above APIs are not performed in current e2e test. By adding test for it, e2e test coverage is improved. /sig testing


Question: import ( "fmt" <del> "reflect" </del> "testing" "time"

What happened: kube-controller-manager panic for What you expected to happen: no panic. How to reproduce it (as minimally and precisely as possible): No idea. I guess it's hard to reproduce. Anything else we need to know?: Found from here: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign cc Should we do something for v1.17? (v1.17 code freeze now, but still have the opportunity to fix) can be removed safely in v1.18, right?
/sig scheduling
Working on this in /sig network /priority critical-urgent /assign
Sorry, I didn't realize it. I should have a search before firing an issue. I recommend adding this issue to your PR. e.g. `
Updated, thanks!
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This seems to be writing to a cached (should be const) service. I think we need to DeepCopy() it before we mutate the Go type. does that sound right? Is this still our best state of the art?
yes, copy before change
we have a tool to catch this (the cache mutation detector), but it got disabled in our test runs two years ago (xref )
yes great catch - I was always convinced this is enabled...


Question: // or has already been reserved.  ErrFull will be returned if there // are no addresses left. func (r *Range) Allocate(ip net.IP) error { <ins> label := r.CIDR() </ins> ok, offset := r.contains(ip) if !ok { <ins> // update metrics clusterIPAllocationErrors.WithLabelValues(label.String()).Inc() </ins> return &ErrNotInRange{r.net.String()} } allocated, err := r.alloc.Allocate(offset) if err != nil { <ins> // update metrics clusterIPAllocationErrors.WithLabelValues(label.String()).Inc() </ins> return err } if !allocated { <ins> // update metrics clusterIPAllocationErrors.WithLabelValues(label.String()).Inc() </ins> return ErrAllocated } <ins> // update metrics clusterIPAllocations.WithLabelValues(label.String()).Inc() clusterIPAllocated.WithLabelValues(label.String()).Set(float64(r.Used())) clusterIPAvailable.WithLabelValues(label.String()).Set(float64(r.Free())) </ins> return nil } // AllocateNext reserves one of the IPs from the pool. ErrFull may // be returned if there are no addresses left. func (r *Range) AllocateNext() (net.IP, error) { <ins> label := r.CIDR() </ins> offset, ok, err := r.alloc.AllocateNext() if err != nil { <ins> // update metrics clusterIPAllocationErrors.WithLabelValues(label.String()).Inc() </ins> return nil, err } if !ok { <ins> // update metrics clusterIPAllocationErrors.WithLabelValues(label.String()).Inc() </ins> return nil, ErrFull } <ins> // update metrics clusterIPAllocations.WithLabelValues(label.String()).Inc() clusterIPAllocated.WithLabelValues(label.String()).Set(float64(r.Used())) clusterIPAvailable.WithLabelValues(label.String()).Set(float64(r.Free())) </ins> return utilnet.AddIPOffset(r.base, offset), nil }

<!-- Feature requests are unlikely to make progress as an issue. Instead, please suggest enhancements by engaging with SIGs on slack and mailing lists. A proposal that works through the design along with the implications of the change can be opened as a KEP: --Add metrics to apiserver tracking the service IP allocator - around how many IP addresses are used(allocated) vs available, and/or if there were any errors allocating service IPs to a service. We want to track if we are approaching the limit of the ServiceClusterIPRange, which is a fixed CIDR and difficult to change. Currently while we can approximate the usage looking at services in the cluster, it is not coming from the source allocator which is within the apiserver. Similar metrics are currently available for the node's ipam allocatior: /area apiserver /sig instrumentation /sig network
/triage accepted /assign I see something like this, gauges for and can you expand on this? a counter incremented each time an IP allocation fails?
that is one option. for contrast the nodeipam metric has an 'allocationtriesper_request' metric - i was imagining something similar to an 'attempts', but with a success:true/false tag/facet so we could compare how often allocations are attempted/how often they fail. I will defer to the experts here as I'm not sure what the status quo for reporting attempts/failures are. I'm not sure if there's any code paths that would cause allocation failures other than usage total (or available <= 0) - but having them separated out like that would be handy in that scenario. (maybe etcd internal allocation is corrupt? or context deadline exceeded)


Question: .PP Possible resources include pods (po), replication controllers (rc), services (svc), nodes, events (ev), component statuses (cs), limit ranges (limits), <del> minions (mi), persistent volumes (pv), persistent volume claims (pvc) </del> <ins> nodes (no), persistent volumes (pv), persistent volume claims (pvc) </ins> or resource quotas (quota). .PP

kubectl 0.17.0: curl http://localhost:8080/api/v1beta3/minions { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "the server could not find the requested resource", "reason": "NotFound", "details": {}, "code": 404 } master3 core # /opt/bin/kubectl get mi master3 core # kubectl 0.16.0: uses: http://localhost:8080/api/v1beta1/minions master3 core # ./kubectl get mi NAME LABELS STATUS 192.168.10.70 <noneReady 192.168.10.71 <noneReady in the same time curl http://localhost:8080/api/v1beta3/nodes works well.
Shall we just remove "minion" from kubectl now and throw an error?
remove "minions"?
Yes, that should be part of . cc
Presumably works?
Assigning who's working on
Ref
Could you please take a look at this?

I know that you can get nodes / a node, but the help doesn't say you can or provide examples: To resolve this issue: the help for this command to include nodes (and any other resources we're missing). we generate the help text in a way that ensures this (and probably the many others that are drifting) don't get out of date? Or at least give us a heuristic as to whether this might be true, even if we have to fix it manually?
Can this be closed?
I'll close this one with ; we want to get rid of the minions (mi) resource name, since it doesn't work for v1beta3.


Question: }, } lockPath14x, lockPath16x := getLockPaths() <ins> // the lockPath14x is a UNIX socket which is cleaned up automatically on close, but the // lockPath16x is a plain file which is not cleaned up. defer os.Remove(lockPath16x) </ins> runner := newInternal(&fexec, ProtocolIPv4, lockPath14x, lockPath16x) var runLock *net.UnixListener

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: files like: are being created, I think from the "unit" tests. What you expected to happen: running tests should not leave files in tree, especially not tracked by git! How to reproduce it (as minimally and precisely as possible): ? I think? Anything else we need to know?: spotted in a PR commit Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig network /assign
I'm also finding an file while working on this locally??? /triage accepted
I suspect , which introduced these lockfile paths and removed deferred removal
is a simple approach to mitigate, but it doesn't solve cluttering the source tree with leaked files. I'm not sure we actually want to take that approach.


Question: import ( "bytes" "fmt" <ins> "net" </ins> "regexp" "strings" "time"

What would you like to be : The test currently only works on Cloud Providers or environments that set the External IP on nodes, because it is using the framework directly to check the IPs of the pods, using a no-snat-test-proxy to check if the Pods are not using SNAT. However, this can be simplified, removing the no-snat-test-proxy Pod and executing the queries directly in a Pod. Why is this needed: The test should work in all environments, and it's failing https://k8s-,%20master&include-filter-by-regex=NoSNAT
/sig network /help wanted /failing test
/remove kind-feature /kind failing-test
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove kind-feature
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I would like to investigate this problem. /assign
I have filed , how could I verify if my patch fixes the problem?
I run them locally with KIND


Question: // if we're using the non-root strategy set the marker that this container should not be // run as root which will signal to the kubelet to do a final check either on the runAsUser // or, if runAsUser is not set, the image UID will be checked. <del> if s.psp.Spec.RunAsUser.Rule == extensions.RunAsUserStrategyMustRunAsNonRoot { </del> <ins> if sc.RunAsNonRoot == nil && s.psp.Spec.RunAsUser.Rule == extensions.RunAsUserStrategyMustRunAsNonRoot { </ins> nonRoot := true sc.RunAsNonRoot = &nonRoot }

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): PodSecurityPolicy RunAsNonRoot Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.7.0-beta, 1.8.0 (HEAD) Environment: Enable PodSecurityPolicy Cloud provider or hardware configuration: gce, probably any OS (e.g. from /etc/os-release): gci Kernel (e.g. ): Install tools: Others: What happened: The PSP admission controller ignores the specified RunAsNonRoot, and overwrites it. What you expected to happen: the admission controller is not supposed to overwrite user specified fields in the podsecuritycontext and securitycontext. How to reproduce it (as minimally and precisely as possible): enable psp, and create a pod with [only] RunAsNonRoot=false specified in a securitycontext. Anything else we need to know:
/sig auth
there are actually 2 bugs here. fails to properly restrict this field when matching against a PSP, which is a latent bug as long as we overwrite the value anyway. overwrites the user-supplied setting for the securitycontext/podsecuritycontext. I suspect the best fix will be to use the API's deep copy function. We are also doing a couple other sketchy things: assigning the "copy" built-in as a variable, and creating an "sc" shadow variable.

I have a half fix in but we should probably fix it in a separate PR, because we may want to patch 1.6.
the effect is that PSP can overly restrict a pod that wanted to run as root, but expressed that as instead of since the workaround is easy (set in your pod if you want to run as root), I don't anticipate backporting the fix.
SGTM. I'll try and get the unit tests in after the GKE summit (on friday).


Question: name:        "unmanaged_zone", provisioner: "kubernetes.io/gce-pd", parameters:  map[string]string{"zone": unmanagedZone}, <del> claimSize:   "1Gi", </del> <ins> claimSize:   "1G", </ins> } sc := newStorageClass(test, ns, suffix) sc, err = c.StorageV1().StorageClasses().Create(sc)

GCE PDs are allocated in chunks of GBs (not GiB) but function incorrectly creates volume in chunks of GiB - cc /sig storage
/kind bug
/assign
/priority important-soon
Hey I submit a pr for this issue. Your suggestions are very important. Thanks.


Question: Name:      "pod-forgiveness-invalid", Namespace: "ns", }, <del> Spec: extendPodSpecwithTolerations(validPodSpec(nil), []api.Toleration{{Key: "node.alpha.kubernetes.io/notReady", Operator: "Exists", Effect: "NoExecute", TolerationSeconds: &[]int64{-2}[0]}}), </del> <ins> Spec: extendPodSpecwithTolerations(validPodSpec(nil), []api.Toleration{{Key: "node.kubernetes.io/not-ready", Operator: "Exists", Effect: "NoExecute", TolerationSeconds: &[]int64{-2}[0]}}), </ins> }, { // docker default seccomp profile ObjectMeta: metav1.ObjectMeta{

Almost all style strings use dashes, rather than javascript names. vs .
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Thanks for notification, a quick fix is submitted.
/cc
The PR here isn't merged yet. Can we make that happen please? We're still in alpha, so should be ok
Should I add the v1.8 milestone to this issue?
Hope so. We are pending in , need approve from
I opened with safe changes. "not-ready" change needs to be discussed more, as it's a breaking change. I believe it's best with fixing that to the time we'll be moving to beta, as we'll need to do another breaking change, i.e. remove 'alpha' from the key.
Aren't we doing this at the 11th hour? Is this going to hit 1.8?
Strings like this that are part of the core API should be part of a design doc that is formally reviewed by an API group. We need to find ways to catch this stuff earlier. We should also have an evolving API style guide that lists stuff like this. /cc
Thanks , let me make some summary: quick fix for 1.8 is , will be merged soon. final fix is , we are collaborating to reach a final solution and it's not target 1.8
Agree with Joe, can someone open a PR to that documents that taint names should follow the naming conventions for annotations?


Question: // get an expired CSR (simulating historical output) server.backdate = 2 * time.Hour <del> server.expectUserAgent = "FirstClient" </del> <ins> server.SetExpectUserAgent("FirstClient") </ins> ok, err := r.RotateCerts() if !ok || err != nil { t.Fatalf("unexpected rotation err: %t %v", ok, err)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: pull-kubernetes-bazel-test Which test(s) are failing: Test_buildClientCertificateManager Testgrid link: Reason for failure: Anything else we need to know:
/remove-kind failing-test /kind flake /sig scheduling
/sig node


Question: } } <del> // test syncNode with instanceV2, same test case with TestGCECondition. func TestGCEConditionV2(t *testing.T) { existingNode := &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ { Key:    cloudproviderapi.TaintExternalCloudProvider, Value:  "true", Effect: v1.TaintEffectNoSchedule, }, }, }, } fakeCloud := &fakecloud.Cloud{ EnableInstancesV2: true, InstanceTypes: map[types.NodeName]string{ types.NodeName("node0"): "t1.micro", }, ProviderID: map[types.NodeName]string{ types.NodeName("node0"): "fake://12334", }, Addresses: []v1.NodeAddress{ { Type:    v1.NodeHostName, Address: "node0.cloud.internal", }, { Type:    v1.NodeInternalIP, Address: "10.0.0.1", }, { Type:    v1.NodeExternalIP, Address: "132.143.154.163", }, }, Provider: "gce", Err:      nil, } clientset := fake.NewSimpleClientset(existingNode) factory := informers.NewSharedInformerFactory(clientset, 0) eventBroadcaster := record.NewBroadcaster() cloudNodeController := &CloudNodeController{ kubeClient:                clientset, nodeInformer:              factory.Core().V1().Nodes(), nodesLister:               factory.Core().V1().Nodes().Lister(), cloud:                     fakeCloud, recorder:                  eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "cloud-node-controller"}), nodeStatusUpdateFrequency: 1 * time.Second, } stopCh := make(chan struct{}) defer close(stopCh) factory.Start(stopCh) factory.WaitForCacheSync(stopCh) w := eventBroadcaster.StartLogging(klog.Infof) defer w.Stop() cloudNodeController.syncNode(context.TODO(), existingNode.Name) updatedNode, err := clientset.CoreV1().Nodes().Get(context.TODO(), existingNode.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("error getting updated nodes: %v", err) } conditionAdded := false for _, cond := range updatedNode.Status.Conditions { if cond.Status == "True" && cond.Type == "NetworkUnavailable" && cond.Reason == "NoRouteCreated" { conditionAdded = true } } assert.True(t, conditionAdded, "Network Route Condition for GCE not added by external cloud initializer") } // This test checks that a node with the external cloud provider taint is cloudprovider initialized and // the GCE route condition is added if cloudprovider is GCE func TestGCECondition(t *testing.T) { existingNode := &v1.Node{ ObjectMeta: metav1.ObjectMeta{ Name:              "node0", CreationTimestamp: metav1.Date(2012, 1, 1, 0, 0, 0, 0, time.UTC), }, Status: v1.NodeStatus{ Conditions: []v1.NodeCondition{ { Type:               v1.NodeReady, Status:             v1.ConditionUnknown, LastHeartbeatTime:  metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), LastTransitionTime: metav1.Date(2015, 1, 1, 12, 0, 0, 0, time.UTC), }, }, }, Spec: v1.NodeSpec{ Taints: []v1.Taint{ { Key:    cloudproviderapi.TaintExternalCloudProvider, Value:  "true", Effect: v1.TaintEffectNoSchedule, }, }, }, } fakeCloud := &fakecloud.Cloud{ EnableInstancesV2: false, InstanceTypes: map[types.NodeName]string{ types.NodeName("node0"): "t1.micro", }, Addresses: []v1.NodeAddress{ { Type:    v1.NodeHostName, Address: "node0.cloud.internal", }, { Type:    v1.NodeInternalIP, Address: "10.0.0.1", }, { Type:    v1.NodeExternalIP, Address: "132.143.154.163", }, }, Provider: "gce", Err:      nil, } clientset := fake.NewSimpleClientset(existingNode) factory := informers.NewSharedInformerFactory(clientset, 0) eventBroadcaster := record.NewBroadcaster() cloudNodeController := &CloudNodeController{ kubeClient:                clientset, nodeInformer:              factory.Core().V1().Nodes(), nodesLister:               factory.Core().V1().Nodes().Lister(), cloud:                     fakeCloud, recorder:                  eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "cloud-node-controller"}), nodeStatusUpdateFrequency: 1 * time.Second, } stopCh := make(chan struct{}) defer close(stopCh) factory.Start(stopCh) factory.WaitForCacheSync(stopCh) w := eventBroadcaster.StartLogging(klog.Infof) defer w.Stop() cloudNodeController.syncNode(context.TODO(), existingNode.Name) updatedNode, err := clientset.CoreV1().Nodes().Get(context.TODO(), existingNode.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("error getting updated nodes: %v", err) } conditionAdded := false for _, cond := range updatedNode.Status.Conditions { if cond.Status == "True" && cond.Type == "NetworkUnavailable" && cond.Reason == "NoRouteCreated" { conditionAdded = true } } assert.True(t, conditionAdded, "Network Route Condition for GCE not added by external cloud initializer") } </del> func Test_reconcileNodeLabels(t *testing.T) { testcases := []struct { name           string

This pertains more for clusters running in GCP but not necessarily in a GKE environment. Clusters that using a CNI supporting overlay such as Calico, do not need explicit routes to be created on the infrastructure level, meaning that the route controller should not be necessary. The following snippet sets the condition for new GCE nodes when a new node is picked up by the node controller. This condition is supposed to be removed later by the route controller in However, as stated above there are cases where the route controller is not needed and should not be running. When Calico is deployed in such a way that the calico-node pods are ignoring any and all taints including the cloudprovider's taint (which we consider reasonable for a system pod such as calico), the above check creates the following race condition: a Calico pod may start and set the node network conditions ( to false) before the node controller reconciles the node. After that the node controller picks up the node and sets the condition to an otherwise healthy node. Leaving aside the race condition itself, I have to ask whether the check is meaningful in the first place. AFAIK, the kubelet in Kubelet only sets the condition for provider, but clusters should already be using kubelet with cloudprovider= For the node controller to not set the condition unnecessarily. An easy way to do it with a custom CCM image is to introduce some delay in the node sync loop, enough for pods such as calico-node to start on that node. No response <details</details<detailsGCP </details<details</details<details</details<details</details<detailsCalico </details
This should be fixed by /cc
/sig cloud-provider /sig network /assign
/triage accepted


Question: "k8s.io/apiserver/pkg/storage/value" ) <del> var ( // The short keepalive timeout and interval have been chosen to aggressively // detect a failed etcd server without introducing much overhead. keepaliveTime    = 30 * time.Second keepaliveTimeout = 10 * time.Second // dialTimeout is the timeout for failing to establish a connection. dialTimeout = 10 * time.Second ) </del> <ins> // The short keepalive timeout and interval have been chosen to aggressively // detect a failed etcd server without introducing much overhead. const keepaliveTime = 30 * time.Second const keepaliveTimeout = 10 * time.Second // dialTimeout is the timeout for failing to establish a connection. // It is set to 20 seconds as times shorter than that will cause TLS connections to fail // on heavily loaded arm64 CPUs (issue #64649) const dialTimeout = 20 * time.Second </ins> func newETCD3HealthCheck(c storagebackend.Config) (func() error, error) { // constructing the etcd v3 client blocks and times out if etcd is not available.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I upgraded kube-apiserver on a Raspberry Pi 3 cluster and after it begins adding the http listeners, the cpu usage goes up on all processors to use every available cycle and all attempts to connect to the api via https (6443) or http (8080) time out. Since I host etcd on the controller nodes, it also becomes unresponsive. If I set GOMAXPROCS=1, it does limit that and prevents etcd from timing out, but all attempts to connect to kube-apiserver still time out. Eventually, the initial IP allocation check times out, causing a fatal error. What you expected to happen: It should idle at about 4%. How to reproduce it (as minimally and precisely as possible): On a raspberry pi 3 running Debian jessie (arm64), I run: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.3 arm64 Cloud provider or hardware configuration: Raspberry Pi 3 OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: n/a Others:
I've tried profiling this, but since it doesn't respond to the http interface, I can't get it that way. I've tried hacking in the ability to write profile data to a file, but the os.Exit in the fatal error log causes the defers to get skipped the the profile file to remain empty or have corrupted data if I do get it to write.
/sig api-machinery
Hello, I nearly problem. Do you use etcd v2? If we use etcd v2, we need "--storage-backend=etcd2" options. But api-server logs don't notice to us.
Nope, etcd 3.2.13 per
/cc
Tested 1.10.5 and 1.11.0... still have the same problem.
Actually, 1.11 is not causing the problem. My test method was flawed. 1.10 is still broken but I can upgrade.

/close
I'm still having the issue with 1.11, i'ld like to know what was your fix
I don't know. It worked fine for a while then I needed to restart that node for unrelated purposes and it failed again so I'm still having the problem, too. /reopen
I've built a binary with go 1.10 and it still has the same problem, fwiw.
says but the server being tested was reverted back to 3.1.11 () for e2e testing due to performance problems. I'll try that next.
running etcd 3.2.18 : Still have the issue
Ok, bisected this trouble back to . PTAL
Reverted and commit on v1.11.0 and have a on arm64:
Built with go1.11beta1 which contains several changes to math/big specifically to address performance issues with arm64. Since these functions are used with encryption, they have a direct affect on establishing TLS communications. That build allowed the TLS connections to establish within the timeout (10s) whereas they were not before. I'll try extending the timeouts and building with go1.10 again and see if the problem can be worked around without using the beta compiler. Part of this may be the fact that I was using RSA certificates instead of ECDSA. If you are, too, you may be able to work around this problem by regenerating your certificates to use ECDSA.


Question: return wc, nil } <ins> // calculateCPUMaximum calculates the maximum CPU given a limit and a number of cpus while ensuring it's in range [1,10000]. func calculateCPUMaximum(cpuLimit *resource.Quantity, cpuCount int64) int64 { cpuMaximum := 10 * cpuLimit.MilliValue() / cpuCount // ensure cpuMaximum is in range [1, 10000]. if cpuMaximum < 1 { cpuMaximum = 1 } else if cpuMaximum > 10000 { cpuMaximum = 10000 } return cpuMaximum } </ins>

When submitting a windows pod using resource limits on a node with more than one processor group, the specified count is incorrect due to the usage of which does not take processor groups into account. For example: node with 96 cores cpu limit 94 returns This results in a CPU maximum of since the range is from [1, ]: Our initial goal for this was to limit a process to not use all available CPU and therefore render the node uncontrollable. A CPU maximum of instead of for that specific case. For the calculation, it should use the CPU count for all available processor groups instead of just the current one. Launch a node with a lot of CPUs, for example and schedule the following pod on it: Then SSH into the node and inspect the running container using . The value will be set to . (which just uses ) is able to correctly find out the number of CPUs available to a system. Using for calculation or incorporating that code might solve this issue. <details</details<detailsAzure in-tree </details<details</details<details</details<detailscontainerd://1.6.10 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
There is already some code which uses GetActiveProcessorCount:
/assign
/triage accepted


Question: reservedSystemCPUs := configureTopologyManagerInKubelet(f, oldCfg, policy, scope, configMap, numaNodes) runTMScopeResourceAlignmentTestSuite(f, configMap, reservedSystemCPUs, policy, numaNodes, coreCount) <ins> }) ginkgo.AfterEach(func() { // restore kubelet config </ins> setOldKubeletConfig(f, oldCfg) <del> deleteStateFile() </del> }) }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-node-kubelet-serial node-kubelet-serial job Which test(s) are failing: Hard to tell, but everything at one point or another. Since when has it been failing: A month or two, by recollection. would indicate several years... ci-kubernetes-node-kubelet-serial | failing_days | 992 Testgrid link: Reason for failure: Hard to tell, some is timeout, but I think timeouts due to test failures. Anything else we need to know: Nothing consistent about the failures that I can tell besides timeouts. This is the only place these serial tests are running.
/sig node
/cc
I will take a look at what happens with the serial lane.
So the kubelet failed to restart because of I think it can be related to the topology manager scope PR, I will dig into the code tomorrow. FYI
It actually looks like it was introduced by: With the same issue reported here: And Alexey has a fix for it here (I haven't reviewed it):
Thanks for the update.
Some of these failures are definitely due to , which is causing tests to fail since pods aren't terminating in time. See e.g.
/assign
Under the latest run, I saw that the kubelet failed to restart because of The problem that one of the topology manager tests failed in the middle and did not remove the state file, I can not guarantee that it a single problem, but let's start from it.


Question: "time" corev1 "k8s.io/api/core/v1" <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/sets"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --my mutating webhook calculates a hash of the , allows reinvocation; no side effects my validating webhook validates the hash of the and fails because the has changed webhook auditing indicates that my webhooks are the only ones processing the upon serializing the spec to JSON for both the mutator and validator, the only change is related to the "kube-api-access-xyz" projected volume, managed by a built-in k8s admission plugin the aforementioned plugin does not appear to mutate the reinvocation context, even through it may mutate a pod (robbing other mutators of the chance to observe the change): it appears that both my hooks see the injected volume, but the validation hook sees the injected SA volume with a "defaultMode" field set, and my mutation hook does NOT see this field set my validation hook should calculate the same hash mutating, validating webhooks are implemented atop controller-runtime v0.8.3, pod objects are being decoded into targets Kubernetes version (use ): 1.21.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g.

): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
cc
interesting... defaults are applied when going from versioned → internal server decodes v1 Pod request → applies defaults → converts to internal Pod -in admission control operates on the internal Pod service account admission injects the kube-api-access volume admission is called internal Pod → converts to v1 Pod → serializes → sends to mutating webhook mutating webhook returns patch patch is applied to v1 Pod JSON resulting v1 Pod JSON is decoded → defaults applied (defaultMode is set on the api-token volume here) → converted to internal Pod webhook admission is called internal Pod → converts to v1 Pod → serializes → sends to validating webhook That means if any built-in admission plugin sets up data that would be modified by applying defaulting to the pod, the data sent to mutating webhooks will be missing those applied defaults until the first mutating webhook that returns a patch that changes the object
Possible solutions: defaults are applied to the internal → versioned object before invoking webhooks built-in admission always populates data matching default values for unset fields
As an aside, I'm not sure what you're using the hashing for, but be aware that hashing is fragile in the presence of defaults and new fields over time. A hash you add to a pod spec in one release might not match a hash computed in a different release, or with different admission in place.
/triage accepted /assign /priority backlog


Question: } if source.ServiceAccountToken.Path == "" { allErrs = append(allErrs, field.Required(fldPath.Child("path"), "")) <ins> } else if !opts.AllowNonLocalProjectedTokenPath { allErrs = append(allErrs, ValidateLocalNonReservedPath(source.ServiceAccountToken.Path, fldPath.Child("path"))...) </ins> } } if projPath := srcPath.Child("clusterTrustBundlePEM"); source.ClusterTrustBundle != nil {

Originally posted by in
Hello, I'm new here. Can I try solving this? it seems like a good first issue
sure, context is the change is needed here: We have to tolerate existing pods / pod specs that fail this validation, so we should only do this additional validation on create or if the existing object passes this validation. To do this: an field to , initialize that field to false, and then conditionally override it to if is non-nil and has a projected token volume path that fails (will have to export the function to check this) , if is false, call the validation on the path:
Hi, I'd like to work on this if there is no one is proceeding on. /assign


Question: configs[CudaVectorAdd] = Config{e2eRegistry, "cuda-vector-add", "1.0"} configs[CudaVectorAdd2] = Config{e2eRegistry, "cuda-vector-add", "2.0"} configs[Dnsutils] = Config{e2eRegistry, "dnsutils", "1.1"} <del> configs[DebianBase] = Config{googleContainerRegistry, "debian-base", "0.4.1"} </del> configs[EchoServer] = Config{e2eRegistry, "echoserver", "2.2"} configs[Etcd] = Config{gcRegistry, "etcd", "3.3.15"} configs[GBFrontend] = Config{sampleRegistry, "gb-frontend", "v6"} configs[GlusterDynamicProvisioner] = Config{dockerGluster, "glusterdynamic-provisioner", "v1.0"} configs[Httpd] = Config{dockerLibraryRegistry, "httpd", "2.4.38-alpine"} configs[HttpdNew] = Config{dockerLibraryRegistry, "httpd", "2.4.39-alpine"} <del> configs[Invalid] = Config{gcRegistry, "invalid-image", "invalid-tag"} </del> configs[InvalidRegistryImage] = Config{invalidRegistry, "alpine", "3.1"} configs[IpcUtils] = Config{e2eRegistry, "ipc-utils", "1.0"} configs[JessieDnsutils] = Config{e2eRegistry, "jessie-dnsutils", "1.0"}

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: jobs primarely, like: Which test(s) are failing: [] Container Runtime blackbox test when running a container with a new image should be able to pull image from [NodeConformance] Since when has it been failing: Since this commit merged: Testgrid link: Reason for failure: The test should always pull the image from (), it is not configurable. Anything else we need to know: /sig testing /sig windows /area conformance This issue was fixed once before:


Question: configuration *atomic.Value lister        admissionregistrationlisters.MutatingWebhookConfigurationLister hasSynced     func() bool <ins> // initialConfigurationSynced stores a boolean value, which tracks if // the existing webhook configs have been synced (honored) by the // manager at startup-- the informer has synced and either has no items // or has finished executing updateConfiguration() once. initialConfigurationSynced *atomic.Value </ins> } var _ generic.Source = &mutatingWebhookConfigurationManager{}

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: Testgrid link: Example pass: Example fail: Reason for failure: Unknown Anything else we need to know: Looks like it's an apiserver test, but I'm not too sure. links to appreciated: <!-- Please see the deflaking doc () for more guidance! --/kind flake
Looks like is flaky too. Example fail:
/sig api-machinery
Thanks for the report. Please report individual failing tests/packages so the responsible sigs can be assigned. The cm/devicemanager issue is tracked in
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
I don't see individual issues yet, but I do see open PRs targeting this issue /remove-lifecycle rotten
closing in favor of specific issues: /close
Closing this issue. <detailsIn response to : closing in favor of specific issues: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: embed = [":go_default_library"], deps = [ "//pkg/api/testing:go_default_library", <ins> "//pkg/apis/core:go_default_library", </ins> "//pkg/scheduler/algorithm:go_default_library", "//pkg/scheduler/algorithm/priorities:go_default_library", "//pkg/scheduler/api:go_default_library",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.

If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.


Question: } } if count == 0 { <del> w.Write(LEVEL_1, "%st%s t%s (%s)n", "*", "*", backendStringer(def), i.describeBackendV1beta1(ns, def)) </del> <ins> w.Write(LEVEL_1, "%st%s t<default>n", "*", "*") </ins> } printAnnotationsMultiline(w, "Annotations", ing.Annotations)

I ran on a fresh minikube with no other services in it and saw this output. Which suggests there is a service in the cluster called (there isn't), and further suggests there is something wrong (there isn't) because there are no endpoints found. I expected to provide me with an accurate description of my Ingress object. This behaviour appears to be hardcoded into the kubectl describe command. Which contains a comment suggesting that this is default behaviour, but no logic to check if the service actually exists before attempting to default to it. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
See also do you know the history here?
not beyond what shows: Looks like an (incorrect) assumption in the describer that the only implementation would behave in this way
/assign
work in progress...
when you get the PR up please link this issue and in it and ping me directly on slack to get a fast review. I agree with what was stated here and in the other issue, that this seems like an unnecessary assumption that should've not been there, we probably should just leave it empty or per print as the value.


Question: import ( "fmt" <ins> "sort" </ins> "strconv" "time"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: is not sent in a stable sort order. this makes node status patch requests larger than needed, and makes debugging nodes harder than necessary. What you expected to happen: stable sort of volumesInUse How to reproduce it (as minimally and precisely as possible): always Anything else we need to know?: nope Environment: n/a
/sig node
- this is a good getting started item.
I've already submitted a PR in . PTAL. Thanks.


Question: name = "go_default_test", srcs = [ "cache_test.go", <ins> "container_hash_test.go", </ins> "helpers_test.go", "ref_test.go", "runtime_cache_test.go",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: During the update from v1.7.1 to v1.8.0 all docker containers were restarted, because container spec hash changed. What you expected to happen: No restart of the docker containers, because they did not change at all. How to reproduce it (as minimally and precisely as possible): get yourself a k8s cluster in v1.7.x update (at least the kubelet) to v1.8.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/area kubelet
I ran into this earlier today. I was able to update the control plane & kube-proxy to 1.8.1 without any problems. Once I upgraded the kubelet though it caused container restarts.
/sig apps /sig api-machinery
the log message one receives is at: The function, that calculates the hashes is at: The hash function seems to not have changed. I suspect changes in the object itself, which is at: One can see: but:
But the new field does not initiated, so the restart seems unreasonable.
the result i tested was that the spew pkg also dumps the new add field(with nil value) to the hasher. is the new field. so, if the container use , then new MountPropagationMode will be included into the hash input, the hash result changed. other new fields are in the same situation.
Do you have any idea about preventing this?
We've hit this as well. Caused a cluster-wide restart of every container.
i also hit this issue as well, when I am upgrading cluster from v1.7.4 to v1.8.5. especially, the kubelet version from v1.7.4 to v1.8.5
Still seems to happen in 1.9.0
I can confirm this. v1.9.0 has introduced a new field attribute for , which is causing the hash value get changed.
It will exist when we use docker. I remember had described it.

There are details in this issues below:
thanks for digging out the old issue! Yes, any new field to the container spec would cause the hash to change. This is not a new issue but new fields are to the container spec rarely (compared with pod spec) so it doesn't happen so often. As discussed in , a new way to generate the hash (e.g., using only updatable fields) or other method for kubelet to examine and compare the state of the container against the spec is needed. Calling this (i.e., container restart) out in the 1.9 release note would be nice.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
/close
Just wondering, why don't we calculate the container hash by since container image is the only mutable field? I know we have to do some refactoring to , but wouldn't it be more reliable?


Question: audienceMatchPolicyFldPath := field.NewPath("issuer", "audienceMatchPolicy") testCases := []struct { <del> name        string in          []string matchPolicy string want        string </del> <ins> name                          string in                            []string matchPolicy                   string want                          string structuredAuthnFeatureEnabled bool </ins> }{ { name: "audiences is empty",

For example, should not let you set discovery URL when the feature gate is not enabled. I believe this only impacts library users, as direct usage in KAS requires the file based config, which is feature gated. /assign enj aramase /milestone v1.32


Question:
v1.Binding

Binding ties one object to another. For example, a pod is bound to a node by a scheduler.

Binding ties one object to another; for example, a pod is bound to a node by a scheduler. Deprecated in 1.7, please use the bindings subresource of pods instead.

 <colgroup>

Forked from We need to update the scheduler (ours and mesos) to use pods/binding instead of /binding and then we can delete /binding.
cc
This is about the top-level resource, not the subresource.
This is where mesos is using binding as top level resource: We need to change that to Resource("pods").Subresource("bindings")
/cc
If we turn this off in 1.2, will that be a problem for you?
Did we announce we were going to remove this API? Don't we need to have a process for taking it away?
Yes it would be problematic unless our scheduler was updated to use whatever was replacing it. AFAICT even the core scheduler is still using /bindings: , Clayton Coleman wrote:
Kicking this can down the road.
We should mark this deprecated.
According to , kube-mesos refactor will use HTTPExtender, so it's a not big issue to kube-mesos. But for k8s, I think scheduler is using API now ().
/assign
/close I'm going to close this; if any comments, please let me know.
We need to actually mark the endpoint deprecated so we can remove it.
OK, got that; need to deprecate it for other user. I think k8s can handle it to use .
, can you help to remove ? it's not related with Mesos anymore. I'll create a PR to deprecate .
Reopening since that PR just a deprecation warning. We still need to remove the API.
wow, agree. , would you help to remove milestone? We will not remove API in 1.7 or 1.8; but I'll trace that.
Done
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
18 still have.
21 still have the resource. Is anyone already working on it or is there a reason why the binding resource still get served?

not entirely sure if i'm right, but from my current assumption it seems that there isn't an implementation for the for pods (xref: ) and we still use the generic resource for pods (). The resource is deprecated since 1.7 but i don't see any further progress :thinking: () dump of an :
what is the progress with this issue? Is it a backlog item that is tracked by sig scheduling?
I'm not aware of this issue. We will put it into our sig meeting to discuss.
we went through this in today's scheduling meeting. We're ok with shifting v1.Binding to pod's subresource. Before acting on it, there are several factors we need to consider: impact to external integrators - either 3rd-party scheduler or scheduler plugins (using extension point). Follow the conventional API deprecation/removal guide/process to ensure a smooth migration. overall volume of efforts. In addition to scheduling efforts, are there any API Server side efforts that need to be tweaked as well? we need to get +1 from
Is there a summary of why moving this API is necessary given that is has been around for quite sometime now? Is it related to RBAC?
/binding is a v1 API, and as such, deletion is not permitted. It can be marked deprecated, and users pointed to pods/binding by documentation as the preferred method, but not removed.
It's more to pay the technical debt. Given the action "binding" is performed against pods, composing a pod/binding instead of v1.Binding makes more sense. Thanks for clarification we cannot remove v1 APIs.
ok, thanks for the clarification, so this is not enabling a new use case, only a cleanup.
Right, and due to the stable API compatibility guarantees, I don't think it's a cleanup we can do. /close
Closing this issue. <detailsIn response to : so this is not enabling a new use case, only a cleanup. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Sorry for the late comment on this but i was AFK for a longer time. I'm not sure if we missed something. Currently there is only the implementation.

via the the API endpoint exists but it refers to the only existing kind (could be checked via ): As there isn't a seperate implementation for IMO we should remove the deprecation warning as we don't have an alternative implementation for that in place. / WDYT?


Question: _, err = recyclerClient.CreatePod(pod) if err != nil { if errors.IsAlreadyExists(err) { <del> glog.V(5).Infof("old recycler pod %q found for volume", pod.Name) </del> <ins> deleteErr := recyclerClient.DeletePod(pod.Name, pod.Namespace) if deleteErr != nil { return fmt.Errorf("failed to delete old recycler pod %s/%s: %s", pod.Namespace, pod.Name, deleteErr) } // Recycler will try again and the old pod will be hopefuly deleted // at that time. return fmt.Errorf("old recycler pod found, will retry later") </ins> } else { return fmt.Errorf("unexpected error creating recycler pod:  %+vn", err) } } <del> defer func(pod *v1.Pod) { glog.V(2).Infof("deleting recycler pod %s/%s", pod.Namespace, pod.Name) if err := recyclerClient.DeletePod(pod.Name, pod.Namespace); err != nil { glog.Errorf("failed to delete recycler pod %s/%s: %v", pod.Namespace, pod.Name, err) } }(pod) </del> <ins> err = waitForPod(pod, recyclerClient, podCh) // In all cases delete the recycler pod and log its result. glog.V(2).Infof("deleting recycler pod %s/%s", pod.Namespace, pod.Name) deleteErr := recyclerClient.DeletePod(pod.Name, pod.Namespace) if deleteErr != nil { glog.Errorf("failed to delete recycler pod %s/%s: %v", pod.Namespace, pod.Name, err) } // Returning recycler error is preferred, the pod will be deleted again on // the next retry. if err != nil { return fmt.Errorf("failed to recycle volume: %s", err) } // Recycle succeeded but we failed to delete the recycler pod. Report it, // the controller will re-try recycling the PV again shortly. if deleteErr != nil { return fmt.Errorf("failed to delete recycler pod: %s", deleteErr) } return nil } </ins> <del> // Now only the old pod or the new pod run. Watch it until it finishes // and send all events on the pod to the PV </del> <ins> // waitForPod watches the pod it until it finishes and send all events on the // pod to the PV. func waitForPod(pod *v1.Pod, recyclerClient recyclerClient, podCh <-chan watch.Event) error { </ins> for { event, ok := <-podCh if !ok {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: If a pod is already running in the default namespace with the expected name (i.e. ), the PV recycle method just assumes it's a recycler pod. However, if it is not a recycler, it could result in the volume being reused without the data being cleared. What you expected to happen: The recycled volume should not be reused until the data has been verifiably deleted. The easiest approach would be to make it an error to not run the recycler pod successfully, i.e. delete these lines: I think that should be safe, since the recycler will retry. Other errors in that method should also be considered fatal, e.g.:
/cc
How about generating a unique name for each recycler pod so that it doesn't matter what existing pods are named?
Could someone from your team pick this up?
Ack
OpenShift runs the recycler pod in a dedicated namespace, not visible to users. Maybe it could be a good idea to do so also in Kubernetes. Anyway, killing the old recycler is probably the best option - it could be forged by user to block recycling and cheat quota. /assign
Agree with running in a dedicated namespace, but it sounds like this feature is just being deprecated, so it might not be worth the effort. Thanks for picking this up.


Question: p.podUpdates[uid] <- workUpdate delete(p.lastUndeliveredWorkUpdate, uid) } else { <del> p.isWorking[uid] = false </del> <ins> // put this line in removeWorker may cause dead lock, so keep reset it here delete(p.isWorking, uid) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As the p.isWorking won't be cleared while pod was deleted, so while creating or deleting pods repeatedly, length of p.isWorking keep increasing length of p.isWorking does not keep increasing creating or deleting pods repeatedly Kubernetes version (use ): 1.19.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign


Question: } cmd.Flags().BoolP("client", "c", false, "Client version only (no server required).") cmd.Flags().BoolP("short", "", false, "Print just the version number.") <del> cmd.Flags().String("output", "", "one of 'yaml' or 'json'") </del> <ins> cmd.Flags().StringP("output", "o", "", "One of 'yaml' or 'json'.") </ins> cmd.Flags().MarkShorthandDeprecated("client", "please use --client instead.") return cmd }

I was confused by the that looks like more as machine readable rather than as human readable. I feel that UX could be better if it will be written in multiple lines and without curly braces/quotes:
Agreed on this one, we should pretty-print this output.
I would like to change that tomorrow if you are not working on it.
I'm not working on it, so feel free to provide a PR. But I wouldn't do anything until we get approval from
cc don't some of the tests rely on this output?
They probably do. There should be a option or something that we can use in test infra.
please have a look at: and related issue: The above is approved and had an lgtm labeled, but removed by the bot because of testing flake.
I think we can close this ?
I don't see that it was fixed. Does the output of was improved?
kubectl now has a option which lets you select between YAML or JSON. However, I believe this is reversed. The more human-friendly output is hidden behind an option and the machine-friendly output is the default. I believe it should be the opposite.


Question: v1 "k8s.io/api/core/v1" kubeletdevicepluginv1beta1 "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1" <ins> "k8s.io/kubernetes/pkg/features" </ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" <ins> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" </ins> "k8s.io/kubernetes/test/e2e/nodefeature" imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api"

In multi test grid: /kind failing-test /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc This is caused by that those test CI didn't enable ResourceHealthStatus. A fix in is to skip it if the FG is disabled.


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2enode import ( "context" "fmt" "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" podutil "k8s.io/kubernetes/pkg/api/v1/pod" "k8s.io/kubernetes/test/e2e/framework" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" "k8s.io/kubernetes/test/e2e/network/common" imageutils "k8s.io/kubernetes/test/utils/image" admissionapi "k8s.io/pod-security-admission/api" ) var _ = common.SIGDescribe("Pod IPs", func() { f := framework.NewDefaultFramework("pod-ips") f.NamespacePodSecurityLevel = admissionapi.LevelRestricted testFinalizer := "example.com/test-finalizer" watchPodIPWhileTerminating := func(ctx context.Context, pod *v1.Pod) { ctxUntil, cancel := context.WithTimeout(ctx, f.Timeouts.PodStart) defer cancel() fieldSelector := fields.OneTermEqualSelector("metadata.name", pod.Name).String() w := &cache.ListWatch{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) { options.FieldSelector = fieldSelector return f.ClientSet.CoreV1().Pods(f.Namespace.Name).Watch(ctx, options) }, } ginkgo.By(fmt.Sprintf("Started watch for pod (%v/%v) to enter terminal phase", pod.Namespace, pod.Name)) _, err := watchtools.Until(ctxUntil, pod.ResourceVersion, w, func(event watch.Event) (bool, error) { if pod, ok := event.Object.(*v1.Pod); ok { found := pod.ObjectMeta.Name == pod.Name && pod.ObjectMeta.Namespace == f.Namespace.Name if !found { ginkgo.By(fmt.Sprintf("Found unexpected Pod (%s/%s) in phase %v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase)) return false, nil } ginkgo.By(fmt.Sprintf("Found Pod (%s/%s) in phase %v, podIP=%v, podIPs=%v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase, pod.Status.PodIP, pod.Status.PodIPs)) if pod.Status.Phase != v1.PodPending { gomega.Expect(pod.Status.PodIP).NotTo(gomega.BeEmpty(), fmt.Sprintf("PodIP not set for pod (%s/%s) in phase %v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase)) gomega.Expect(pod.Status.PodIPs).NotTo(gomega.BeEmpty(), fmt.Sprintf("PodIPs not set for pod (%s/%s) in phase %v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase)) } // end the watch if the pod reached terminal phase return podutil.IsPodPhaseTerminal(pod.Status.Phase), nil } ginkgo.By(fmt.Sprintf("Observed event: %+v", event.Object)) return false, nil }) framework.ExpectNoError(err, "failed to see event that pod (%s/%s) enter terminal phase: %v", pod.Namespace, pod.Name, err) ginkgo.By(fmt.Sprintf("Ended watch for pod (%v/%v) entering terminal phase", pod.Namespace, pod.Name)) } ginkgo.Context("when pod gets terminated", func() { ginkgo.It("should contain podIPs in status for succeeded pod", func(ctx context.Context) { podName := "pod-ips-success-" + string(uuid.NewUUID()) podSpec := e2epod.MustMixinRestrictedPodSecurity(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Name:    podName, Image:   imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{"sh", "-c"}, Args: []string{` sleep 1 exit 0 `, }, }, }, }, }) ginkgo.By(fmt.Sprintf("creating the pod (%v/%v)", podSpec.Namespace, podSpec.Name)) podClient := e2epod.NewPodClient(f) pod := podClient.Create(ctx, podSpec) watchPodIPWhileTerminating(ctx, pod) ginkgo.By(fmt.Sprintf("getting the terminal state for pod (%v/%v)", podSpec.Namespace, podSpec.Name)) pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(ctx, podName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get the final state for pod (%s/%s)", pod.Namespace, pod.Name) gomega.Expect(pod.Status.Phase).To(gomega.Equal(v1.PodSucceeded), fmt.Sprintf("Non-terminal phase for pod (%s/%s): %v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase)) gomega.Expect(pod.Status.ContainerStatuses[0].State.Terminated).ShouldNot(gomega.BeNil()) gomega.Expect(pod.Status.ContainerStatuses[0].State.Terminated.ExitCode).Should(gomega.Equal(int32(0))) }) ginkgo.It("should contain podIPs in status for failed pod", func(ctx context.Context) { podName := "pod-ips-crash-" + string(uuid.NewUUID()) podSpec := e2epod.MustMixinRestrictedPodSecurity(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Name:    podName, Image:   imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{"sh", "-c"}, Args: []string{` exit 42 `, }, }, }, }, }) ginkgo.By(fmt.Sprintf("creating the pod (%v/%v)", podSpec.Namespace, podSpec.Name)) podClient := e2epod.NewPodClient(f) pod := podClient.Create(ctx, podSpec) watchPodIPWhileTerminating(ctx, pod) ginkgo.By(fmt.Sprintf("getting the terminal state for pod (%v/%v)", podSpec.Namespace, podSpec.Name)) pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(ctx, podName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get the final state for pod (%s/%s)", pod.Namespace, pod.Name) gomega.Expect(pod.Status.Phase).To(gomega.Equal(v1.PodFailed), fmt.Sprintf("Non-terminal phase for pod (%s/%s): %v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase)) gomega.Expect(pod.Status.ContainerStatuses[0].State.Terminated).ShouldNot(gomega.BeNil()) gomega.Expect(pod.Status.ContainerStatuses[0].State.Terminated.ExitCode).Should(gomega.Equal(int32(42))) }) ginkgo.It("should contain podIPs in status for during termination", func(ctx context.Context) { podName := "pod-ips-terminating-" + string(uuid.NewUUID()) podSpec := e2epod.MustMixinRestrictedPodSecurity(&v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:       podName, Finalizers: []string{testFinalizer}, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Name:    podName, Image:   imageutils.GetE2EImage(imageutils.BusyBox), Command: []string{"sh", "-c"}, Args: []string{` sleep 9999999 & PID=$! _term() { kill $PID echo "Caught SIGTERM signal!" } trap _term SIGTERM touch /tmp/trap-marker wait $PID exit 42 `, }, ReadinessProbe: &v1.Probe{ PeriodSeconds: 1, ProbeHandler: v1.ProbeHandler{ Exec: &v1.ExecAction{ Command: []string{"/bin/sh", "-c", "cat /tmp/trap-marker"}, }, }, }, Lifecycle: &v1.Lifecycle{ PreStop: &v1.LifecycleHandler{ Exec: &v1.ExecAction{ Command: ExecCommand(podName, execCommand{ Delay: 1, }), }, }, }, }, }, }, }) ginkgo.By(fmt.Sprintf("creating the pod (%v/%v)", podSpec.Namespace, podSpec.Name)) podClient := e2epod.NewPodClient(f) pod := podClient.Create(ctx, podSpec) ginkgo.By(fmt.Sprintf("set up cleanup of the finalizer for the pod (%v/%v)", f.Namespace.Name, pod.Name)) ginkgo.DeferCleanup(e2epod.NewPodClient(f).RemoveFinalizer, pod.Name, testFinalizer) ginkgo.By(fmt.Sprintf("Waiting for the pod (%v/%v) to be running and the SIGTERM trap is registered", pod.Namespace, pod.Name)) err := e2epod.WaitTimeoutForPodReadyInNamespace(ctx, f.ClientSet, pod.Name, f.Namespace.Name, f.Timeouts.PodStart) framework.ExpectNoError(err, "Failed to await for the pod to be running: %q", pod.Name) ginkgo.By(fmt.Sprintf("Deleting the pod (%v/%v) to trigger termination", pod.Namespace, pod.Name)) err = e2epod.NewPodClient(f).Delete(ctx, pod.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "Failed to delete the pod: %q", pod.Name) watchPodIPWhileTerminating(ctx, pod) ginkgo.By(fmt.Sprintf("getting the terminal state for pod (%v/%v)", podSpec.Namespace, podSpec.Name)) pod, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(ctx, podName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get the final state for pod (%s/%s)", pod.Namespace, pod.Name) gomega.Expect(pod.Status.Phase).To(gomega.Equal(v1.PodFailed), fmt.Sprintf("Non-terminal phase for pod (%s/%s): %v", pod.ObjectMeta.Namespace, pod.ObjectMeta.Name, pod.Status.Phase)) gomega.Expect(pod.Status.ContainerStatuses[0].State.Terminated).ShouldNot(gomega.BeNil()) gomega.Expect(pod.Status.ContainerStatuses[0].State.Terminated.ExitCode).Should(gomega.Equal(int32(42))) }) }) }) </ins>

My pod completes (success or failure). The IP temporarily is removed on one of the status updates. The IP is persistent . Apply: Watch: The initial 'succeeded' should have the IP still. On older versions: Full pod state, from 1.27: <details</detailsPrior discussion many, see above Tested on kind Linux <details</details<details</details<details</details
/sig network maybe Relevant PRs in 1.27:
cc
/priority important-soon
is about endpointslices, so no related seems related , ping
I can confirm this commit specifically is the culprit (when I build and test the parent commit it works ok): For now I don't know the reason, cc in case you have some ideas. EDIT: when commenting out these it works as before. I suspect this is because the does not have the information which is determined in
I've opened a PR with a fix. PTAL: Additionally, it turns out that the bug causes one more unnecessary request with status update.
This also relates to sig-node: /sig node We talked about this in the SIG Network meeting, looks like you are taking a look: /assign Please let us know what you find, and please let us know if you need any support!
Thank you! I think I have got to the bottom and summarized the findings in the PR description: I would love to move forward with the review :)
the question from SIG Node meeting was - what is the impact of it besides the flickering IP on status. This may help justify the cherry-pick.
do you have some additional insights about the impact?
In our usage we had to handle events on a pod informer differently to account for the fact we may read it in an invalid state if we read at the wrong time. So for us technically it hsa no impact since we worked around it, for others it could be quite severe potentionally
Thanks! Just FYI as demonstrated by experiments on you could also face this situation in case of kubelet restart. So, probably good you put this workaround in place anyway.
/triage accepted People affected by this regression likely will be affected by kubelet restart in a very precise moment. But it is a low probability event so I would advocate to fix the regression by backporting this fix instead of forcing consumers to work around this behavior.


Question: t.Run(test.name, func(t *testing.T) { nodes := makeLabeledNodeList(labeledNodes) snapshot := cache.NewSnapshot(test.pods, nodes) <del> ctx := context.Background() </del> <ins> ctx, cancel := context.WithCancel(context.Background()) defer cancel() </ins> informerFactory, err := populateAndStartInformers(ctx, test.rcs, test.rss, test.services, test.sss) if err != nil { t.Errorf("error creating informerFactory: %+v", err)

The ctx object () is context.Background(). Thus, its Done() method returns a nil channel. Later, the nil channel is used to call and as their argument. The goroutines created in the two functions are leaked. No goroutine leaks in TestSelectorSpreadScore(). Run TestSelectorSpreadScore() No response all versions after <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling (It sounds like leas in unit tests, which may be negligible. Does it cause some test failures?)
the test can pass.
/assign
/unassign /assign


Question: apiVersion: v1 kind: Service metadata: <del> name: coredns </del> <ins> name: kube-dns </ins> namespace: kube-system labels: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> kubernetes.io/cluster-service: "true" addonmanager.kubernetes.io/mode: Reconcile kubernetes.io/name: "CoreDNS" spec: selector: <del> k8s-app: coredns </del> <ins> k8s-app: kube-dns </ins> clusterIP: __PILLAR__DNS__SERVER__ ports: - name: dns

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-autoscaling]sDNSshorizontalsautoscalings[DisabledForLargeClusters]skube-dns-autoscalersshouldsscaleskube-dnsspodssinsbothsnonfaultysandsfaultysscenarios$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig autoscaling
/unsig cli
Looks like the test is searching specifically for a deployment labeled "kube-dns". There are several references to "kube-dns" scattered around it, actually.
Right now the tool "dns-horizontal-autoscaller" seems dedicated for kube-dns. This tools is deployed as "add-on" when enabled by this ENV variable : I guess we should have the definition of another dns-horizontal-autoscaller, dedicated to coredns (unless we generate the yaml file). and the choice of deployment is based on the ENV variable . see
Currently, the translation of the deployment of "dns-horizontal-autoscaller" for CoreDNS is in progress by It is expected that the corresponding e2e test will work as is (as the artifacts are selected by labels, which are the same for CroeDNS and kube-dns).


Question: } } <ins> if !groupsSpecified && username != user.Anonymous { // When impersonating a non-anonymous user, if no groups were specified // if neither the system:authenticated nor system:unauthenticated groups are explicitly included, // include the system:authenticated group in the impersonated user info found := false for _, group := range groups { if group == user.AllAuthenticated || group == user.AllUnauthenticated { found = true break } } if !found { groups = append(groups, user.AllAuthenticated) } } </ins> newUser := &user.DefaultInfo{ Name:   username, Groups: groups,

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): impersonate system:authenticated Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): Centos7 Kernel (e.g. ): Install tools: custom Others: Using RBAC What happened: Impersonating a user is not setting group which causes impersonate to fail What you expected to happen: If a user is in the tokens file they and they are impersonated the group should be for authz. How to reproduce it (as minimally and precisely as possible): sure group has cluster role a test user via token file a user with impersonate access submit a command via kubectl That request will fail with 403 response even though the user is in the group. Anything else we need to know: cc
We can probably be smart and put the authenticated group adder around the result of the impersonation filter. you see any issues with that?
We provide the group field to provide this kind of information. I'm not sure I would expect my request to the server about a particular user/group tuple to be evaluated for a different user/group tuple.
for impersonation, we already add groups when known (service accounts). doing so for the system:authenticated/system:unauthenticated groups makes sense to me
Only when the groups aren't specified, right? If memory serves, that's how we did it downstream where we can make group determination.