Question: return testEndpointReachability(externalIP, sp.Port, sp.Protocol, execPod) } <del> func testReachabilityOverNodePorts(nodes *v1.NodeList, sp v1.ServicePort, pod *v1.Pod, clusterIP string) error { </del> <ins> func testReachabilityOverNodePorts(nodes *v1.NodeList, sp v1.ServicePort, pod *v1.Pod, clusterIP string, externalIPs bool) error { </ins> internalAddrs := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) <del> externalAddrs := e2enode.CollectAddresses(nodes, v1.NodeExternalIP) </del> isClusterIPV4 := utilsnet.IsIPv4String(clusterIP) for _, internalAddr := range internalAddrs {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We ran sonobuoy on one of our clusters running on AWS and it reported two failed tests: This happened because the pod employed to conduct the tests wasn't able to reach the service using the node's public IP address: What you expected to happen: The security groups on that cluster don't allow for connections to arbitrary ports through the public IP address. The tests should still succeed because that configuration is seen as a security feature. How to reproduce it (as minimally and precisely as possible): Spin up a Kubernetes cluster on AWS EC2 instances, configure SGs to only allow traffic to the instances from other instances (or from an ELB). Run sonobuoy. Anything else we need to know?: Environment: AWS Kubernetes version (use ): 1.16.8 Cloud provider or hardware configuration: AWS OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area conformance /sig network
related:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
I am also seeing this issue when running conformance testing. The nodes in our test cluster (where we run conformance tests) have public IP addresses, but nearly all ports all firewalled. We are also running in AWS. These tests should only rely on the internal IP address of the nodes.
/assign
Yes, I have seen this before, that if the nodes have an external address it will use that one, which means the firewall must be open for that. If the nodes do not have an external IP then the test will pass just fine. It's not as much a bug as an inconvenient implementation. But we can treat it as a bug and fix it. The intent of the test is to make sure the service is plumbed properly.
/remove-triage unresolved
Good point. However, we only assign an external IP address to be able to SSH into the host and thus block all other ports.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm seeing the same issue, though I noticed that it happens only on one node in the whole cluster. Even after re-creating a cluster from scratch, the test always fails only on one specific node. However this node is identical to others, but the test succeed on rest of the nodes.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
In some user scenarios, the external ip nodeport is for access from APPs outside this cluster/zone. However, in most scenarios, external IP is just for SSH or special use(Ingress Gateway, etc.) Different users have different proposals. Why not disable e2e tests checking on external IP with a flag? Will send a PR for this.
That's not all that useful for those running conformance tests. Would be better to change it to always use the internal IP, we don't need the external access for conformance purposes.
/assign
agree, also most of the network tests now run from the cluster, no need to reach externalIPs please tag me in the PR
So kubernetes applications should not expect to be able to access them with an external node address? FWIW we also require upstream / public internet access in the DNS tests, so needing somewhat unrestricted networking to test the cluster behavior is not new.
I'm not extremely familiar with the conformance tests. Are the DNS tests you are referring to about being able to access public services from inside of the cluster (egress)? Because our problem is that our default Kubernetes deployment doesn't allow ingress other than SSH and we haven't seen any problems with the DNS tests so far when checking our product for conformance. That's why I like approach of disabling that specific test with a flag.

Maybe that could also be generalized to let all conformance tests know this is a locked-up cluster with basically no ingress. But that may be a matter of a subsequent PR/issue.
Disabling the tests with a flag is not really an acceptable approach for conformance. The whole point of the conformance program is to validate the Kubernetes implementation against a standard set of tests across all vendors etc. If we don't think access like this is a standard Kubernetes behavior applications should expect anymore, the tests should be removed from conformance and we can consider instead running them just to test the Kubernetes project but not in conformance by moving them to a new test. IMHO we usually only increase the scope of conformance behavior though, not decrease, and it's expected that some users will have more secure clusters that fail conformance (for example we test some privileged behaviors IIRC) ...
After reading comment, I agree that the test should just be removed. However, I'd like to first understand the reasoning behind it being there in the first place. Maybe there's a good reason that such "secured cluster" aren't deemed compliant?!
so, the next step is to review the conformance definition for nodeport and modify the test accordingly Node.Status.Addresses are not a great thing, bad semantics, ... this KEP and this PR summaries perfectly why I personally, would move ExternalIP from Conformance entirely until it has a clear definition
So we should remove it now and then add a new test if needed after a conclusion from the discussion in the KEP. Edited. I may misunderstand. A proposal is to move those external addr NodePort test from conformance to normal e2e test cases In Conformance Case, don't check external ip's NodePort. In e2e test case, check them.
My point is that Conformance folks (cc ) should decide if they want to consider ExternalIPs as part of the conformance definition, I pointed to some issues related to the topic here sig-network will keep testing externalIPs, are part of the API and we need to verify them, Conformance has a narrower scope
The fact that these tests are using ExternalIPs does not need to be in conformance and in fact really shouldn't be. But we still need the tests working against InternalIPs for conformance. It's more of an historical artifact that they use ExternalIPs than an intentional choice.

So, agree with that sig-network should test them, but we should remove the use of the external IPs from conformance, and leave in the tests but against the internal ips only. I resolved this, for example, in GKE conformance runs by using a private GKE cluster that does not give the nodes external IPs.


Question: }{ { parent:   ss1, <del> revision: ss1Rev1, </del> <ins> revision: ss1Rev2, </ins> }, }, rename: true,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT: What happened: When a statefulset object is created, a lot of controllerrevisions are created at the same time. the controllerrevisions have differenct hash value but the same Data.Raw. How to fix it: two controllerrevisions are seen as the same , when [ControllerRevisionHashLabel] , Controllerrevision.Data.Raw , Controllerrevision.Data.Object are strictly the same // this line shoud be to fix the bug like below: Anything else we need to know?: Environment: Kubernetes version (use ): 1.7.8 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I would help fix it
/kind bug /sig apps
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
StatefulSet uses the following code to construct its history object (controllerrevision): However, collisionCount shouldn't be blindly increased when there's name collision. Because controller uses a cache store, the name collision could be due to a controllerrevision created but not yet observed by the controller. The right way to handle it is to first check if the controllerrevision with the same name has the same content as the one we're going to create, like how DaemonSet controller handles it:


Question: if hc != nil { hcNames = append(hcNames, hc.Name) } <del> if err := gceCloud.DeleteExternalTargetPoolAndChecks(loadBalancerName, gceCloud.Region(), hcNames...); err != nil && </del> <ins> if err := gceCloud.DeleteExternalTargetPoolAndChecks(loadBalancerName, region, hcNames...); err != nil && </ins> !IsGoogleAPIHTTPErrorCode(err, http.StatusNotFound) { retErr = fmt.Errorf("%vn%v", retErr, err) }

In federation tests, if we observe the build logs. we see cleaning up LoadBalancer from cloud provider seems not working. It has 2 effects: delays the tests by waiting to delete but always returns error. so we waste like 5 minutes. maybe we are leaking the LoadBalancer in the project.
we can see these errors in every build. but just for reference here are some build logs https://k8s-
cc
All the errors in logs are . It doesn't necessarily indicate we are leaking resource, just now we must have clusterID initialized before calling EnsureLoadBalancerDeleted(). Would be better to not call this function directly in e2e test :)
Federation cleanup should use the same implementation as non-federated e2e test:
function looks like only specific to GCE. in future we would run the e2e tests in AWS too. it would be better to use generic functions if any.
using is working for us. here is the test logs https://k8s-
EnsureLoadBalancerResourcesDeleted() actually cleans up nothing, it only lists forwarding rules and probably logging :)


Question: } wg.Wait() }) <ins> ginkgo.It("should mount multiple PV pointing to the same storage on the same node", func() { // csi-hostpath driver does not support this test case. In this test case, we have 2 PV containing the same underlying storage. // during the NodeStage call for the second volume, csi-hostpath fails the call, because it thinks the volume is already staged at a different path. // Note: This is not an issue with driver like PD CSI where the NodeStage is a no-op for block mode. if pattern.VolMode == v1.PersistentVolumeBlock { e2eskipper.Skipf("skipping multiple PV mount test for block mode") } init() defer cleanup() l.testCase.PvCheck = func(claim *v1.PersistentVolumeClaim) { MultiplePVMountSingleNodeCheck(l.cs, f.Timeouts, claim, l.config.ClientNodeSelection) } _, clearProvisionedStorageClass := SetupStorageClass(l.testCase.Client, l.testCase.Class) defer clearProvisionedStorageClass() l.testCase.TestDynamicProvisioning() }) </ins> } // SetupStorageClass ensures that a StorageClass from a spec exists, if the StorageClass already exists

When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.


Question: # if the url belongs to GCS API we should use oauth2_token in the headers curl_headers="" if { [[ "${KUBERNETES_PROVIDER:-gce}" == "gce" ]] || [[ "${KUBERNETES_PROVIDER}" == "gke" ]] ; } && <del> [[ "$url" =~ ^https://storage.googleapis.com.* ]] && valid-storage-scope ; then curl_headers="Authorization: Bearer $(get-credentials)" </del> <ins> [[ "$url" =~ ^https://storage.googleapis.com.* ]]; then curl_headers="Authorization: Bearer $(gcloud auth print-access-token)" </ins> fi curl ${curl_headers:+-H "${curl_headers}"} -fL --retry 3 --keepalive-time 2 "${url}" -o "${download_path}/${file}" elif [[ $(which wget) ]]; then

Many jobs are failing because of Some of the job logs: /priority critical-urgent /cc /cc /milestone v1.18 /sig release
Also, all the jobs report in their log the following: Do we have to take any action to change that?
After some investigation, it seems that file contains instead of
Switching from to podutils should be done by the job owners - this is out of scope for releas team (strictly speaking). And the message is just a warning, should not affect the job itself.
is this related to the workload identity switch yesterday?
Maybe? get- does seem to pass auth headers, possibly this part isn't working correctly? These jobs explicitly mount a service- though so I wouldn't expect workload identity to change things but I could be wrong...
Yes, this will not work until we attach a kubernetes service account to the job that uses workload identity:
Ideally that command should become (and let it decide how to best authenticate) rather than assuming the script runs on a VM
/assign
sent
/reopen doesn't seem to solve the issue. The jobs are still failing because of this reason. Do we need any more actions after ????
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't agree, the definitely helped in case of e2e scalability jobs, our presubmits and continuous jobs started passing again, e.g. https://k8s-
Looks green?
Only the 1.18 jobs are green. We need to cherry-pick the fix to the other branches as well. I will keep this issue open.
For some reason I thought kubetest always used the copy from HEAD.
in the case that it didn't we'll need this cherry picked to all active branches :|
I thought the same, but indeed the fixed only 1.18 and master jobs. Older release scalability jobs are still broken ! It's a bit confusing TBH. I thought that get- is always downloaded by kubetest from which IIUC points to the master branch.

Alternatively, get- could've been embedded in the kubekins image, but if that was the case then couldn't have fixed our 1.18/master tests without kubekins version update (which didn't happen). Another options is that kubetest downloads get- from proper release branches, but I couldn't find such logic in the kubetest code. would you mind explaining how it works?
There are a lot of jobs that are failing because of this. - master - 1.18 7 jobs in 1.17 branch 5 jobs in 1.16 branch 5 jobs in 1.15 branch Can you please take a look?
/sig testing
Ping, any updates?
filed a number of cherry picks for the script
Closing... The issue has been resolved. Thanks, /close
Closing this issue. <detailsIn response to : Closing... The issue has been resolved. Thanks, /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: function run-gcloud-compute-with-retries { RETRIES="${RETRIES:-3}" for attempt in $(seq 1 ${RETRIES}); do <del> local -r gcloud_result=$(gcloud compute "$@" 2>&1) local -r ret_val="$?" </del> <ins> # We don't use 'local' to declare gcloud_result as then ret_val always gets value 0. gcloud_result=$(gcloud compute "$@" 2>&1) || local ret_val="$?" </ins> echo "${gcloud_result}" <del> if [[ "${ret_val}" -ne "0" ]]; then </del> <ins> if [[ "${ret_val:-0}" -ne "0" ]]; then </ins> if [[ $(echo "${gcloud_result}" | grep -c "already exists") -gt 0 ]]; then if [[ "${attempt}" == 1 ]]; then echo -e "${color_red}Failed to $1 $2 $3 as the resource hasn't been deleted from a previous run.${color_norm}" >& 2

I'm seeing the following in my logs: There was an errors, and we silently skipped it.
Good catch! Thanks for pointing.


Question: return true, nil } <ins> // InstanceExists returns true if the instance with the given provider id still exists and is running. // If false is returned with no error, the instance will be immediately deleted by the cloud controller manager. func (g *Cloud) InstanceExists(ctx context.Context, node *v1.Node) (bool, error) { providerID := node.Spec.ProviderID if providerID == "" { var err error if providerID, err = g.InstanceID(ctx, types.NodeName(node.Name)); err != nil { return false, err } } return g.InstanceExistsByProviderID(ctx, providerID) } // InstanceMetadata returns metadata of the specified instance. func (g *Cloud) InstanceMetadata(ctx context.Context, node *v1.Node) (*cloudprovider.InstanceMetadata, error) { timeoutCtx, cancel := context.WithTimeout(ctx, 1*time.Hour) defer cancel() providerID := node.Spec.ProviderID if providerID == "" { var err error if providerID, err = g.InstanceID(ctx, types.NodeName(node.Name)); err != nil { return nil, err } } _, zone, name, err := splitProviderID(providerID) if err != nil { return nil, err } region, err := GetGCERegion(zone) if err != nil { return nil, err } instance, err := g.c.Instances().Get(timeoutCtx, meta.ZonalKey(canonicalizeInstanceName(name), zone)) if err != nil { return nil, fmt.Errorf("error while querying for providerID %q: %v", providerID, err) } addresses, err := nodeAddressesFromInstance(instance) if err != nil { return nil, err } return &cloudprovider.InstanceMetadata{ ProviderID:    providerID, InstanceType:  lastComponent(instance.MachineType), NodeAddresses: addresses, Zone:          zone, Region:        region, }, nil } </ins> // InstanceID returns the cloud provider ID of the node with the specified NodeName. func (g *Cloud) InstanceID(ctx context.Context, nodeName types.NodeName) (string, error) { instanceName := mapNodeNameToInstanceName(nodeName)

For external cloud provider - Kubelet setting the TaintExternalCloudProvider taint: The results in calling function in cloud node-controller: This (in a healthy case) sends 4 GCE API calls to: node addresses: (as part of getNodeModifier call) instance type: (as part of getNodeModifiers call) instance: (as part of updateNodeAddresses call) node addresses (again): (as part of updateNodeAddresses call) All those requests are read requests and could be replaced by a single call. We could do that by introducing a simple struct like: and adding a method (or rather replacing the existing InstanceTypeByProviderID which is used only for this purpose) to Instances cloud-provider interface; With that change, we would be able to replace all that 4 calls with just a single API call. This would be useful especially for creating large clusters (like 5k nodes), which may saturate the quota for API calls for quite some time. This is quite significant size of refactoring, though it should be relatively mechanical task to do. /sig cloud-provider /sig scalability /priority important-longterm /help-wanted - FYI
The InstanceMetadataByProviderID would be to every cloud provider, such as aws, azure, etc. right ?
/assign Hi , if you're interested in it, I can work with you.
Yes - we need to do that (which is why I wrote it's quite significant refectoring). It also makes reviews harder, because we will need to ask maintainers of all cloud-providers. So it would be good to separate changes to different cloud providers into separate commits.
I have one question: and first query by providerID, if failed then query by nodeName. While the logic in is converse(if providerID is nil, get providerID by nodeName, then ensure its existence by providerID). If we merge these three method into , how to deal with the query sequence of nodeName and providerID?
I bet it's a consequence of different people adding different code paths. Those conceptually seem to be the same to me. Unifying this will be additional benefit from this change imho. So I would say that choose whatever is easier to implement.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I think and are the only PRs left for this. One other optimization that can be made is to cache the instance metadata but I think that's a provider specific implementation
Plus enabling it for GCE if we decide to remove it from the first PR. But yeah, in general I agree.
- can you please address ?
Yes, I have seen that.
I think this issue has been resolved, feel free to reopen it. /close
Closing this issue. <detailsIn response to : I think this issue has been resolved, feel free to reopen it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // Empty if the Pod has no claims. claims []*resourcev1alpha2.ResourceClaim <del> // The AvailableOnNodes node filters of the claims converted from the // v1 API to nodeaffinity.NodeSelector by PreFilter for repeated // evaluation in Filter. Nil for claims which don't have it. availableOnNodes []*nodeaffinity.NodeSelector </del> // The indices of all claims that: // - are allocated // - use delayed allocation

The node selectors gets parsed by in for each node. That work can be done once in . Better performance.
/sig node /sig scheduling /triage accepted /priority backlog
I can try this one. /assign


Question: if mounter == nil { return false, fmt.Errorf("mounter was not set for a filesystem volume") } <del> if isNotMount, mountCheckErr = mounter.IsLikelyNotMountPoint(mountPath); mountCheckErr != nil { </del> <ins> if isNotMount, mountCheckErr = mount.IsNotMountPoint(mounter, mountPath); mountCheckErr != nil { </ins> return false, fmt.Errorf("could not check whether the volume %q (spec.Name: %q) pod %q (UID: %q) is mounted with: %v", uniqueVolumeName, volumeName,

When a pod is marked as deleted while kubelet is down / being restarted, newly started kubelet does not clean up CSI filesystem volumes of the pod. Newly started kubelet tries to reconstruct the volume using CSI's function. This part looks working, CSI volume plugin loads its json file. But then VolumeManager checks if the volume is still mounted in directory. There are two issues: does not require volumes to be presented as mounts. They can be just directories with files on them. This will be case of the most of in-line volumes. Kubelet checks the right directory given by
/sig storage related to cc
Maybe a stupid idea... Reconciler sync has found a directory in . Reconciler has no way how to check that CSI volume (or any other volume) is really mounted / published there. Depending on DSW: The volume is in DSW -from comment here it seems that the SetUp (=NodePublish) will be called on the volume: The volume is not in DSW and it will be to ASW, so it's unmounted / unpublished in the next sync: In both cases, we can expect that the volume plugin / CSI driver is idempotent and SetUp() won't do anything if the volume is already set up / TearDown() will not do anything if the volume has been torn down already. Are our volume plugins really idempotent? IMO they are. We perhaps don't need to check for presence of mount point anywhere, just presence of directory should be enough to reconstruct the volume and set it up / tear it down. Adding to the loop.
Filed , PTAL
I'm curious how the reconstruction e2e tests passed
https://k8s-*force
They start kubelet with a pod already deleted. If the pod still exists, volume reconstruction can see it in DSW and does nothing when fails: [Reconstruction fails because it checks for a mount in a wrong directory.] In this case, kubelet waits for the volume to appear in VolumesInUse before calling . The pod is deleted before that and kubelet then just deletes DSW entry and the volume is never cleaned up.
They pass because their container does not handle SIGTERM and it takes 30 seconds to kill them. Kubelet has enough time to the volume during reconciliation.

It turns out that I tested with broken version of and reconstruction is broken only for ephemeral volumes that don't mount.
So the conclusion is that we only need to fix for ephemeral volumes? Thanks!
Yes, sorry again about the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle rotten
/lifecycle frozen
From this issue surfaced through e2e tests using the hostPath CSI driver. The cause is (which is used in for reconstruction) - it doesn't play well with volumes that bind mount from a directory instead of a device.
To work around this bug here, volume life cycle checking was disabled for all tests, not just the subpath test: This check is useful and should be enabled again.
Hey everyone! I arrived at this conversation by investigating on issue (seems to me like a duplicate of ). I'm wondering if there's something I can help with? I'm still a bit new to the codebase, though. I was able to reproduce the problem by using the following ginkgo focus command in the e2e framework: And do see that if we manually add the --check-volume-lifecycle=false to the statefulset/csi-hostpathplugin tests will pass correctly. I also see that during the process, something in the cleanup function doesn't correctly delete the pv's after the tests are done, so they remain alive if we execute a kubectl get pv command.
I think I found a bit of more info on the issue probably could be of some help. I think that the volume is not being correclty marked as unmounted inside in GenerateUnmountVolumeFunc. I'm not sure why but I'll continue to investigate on it. I think this error is coming from DeletePodFromVolume in
I a couple of extra debug lines to that lines in the file, and can see that when DeletePodFromVolume tries to see if the volume exists, it doesn't find it in the asw.attachedVolumes struct, like so: Probably we should search for the name of the attached volume differently for ephemeral volumes?
/assign
There are two problems: 1) The IsLikelyNotMountPoint() call that , which is known not to work for bind mounts. The mountpoint never gets found, so volume reconstruction fails with "... is not mounted", the PVC never never gets to the list of attached volumes, and that's why we end up hitting this error: 2) CSI inline volumes hit a different error (in addition to the first problem). The call path looks like this (starting in ): Inline volumes do not have a PV spec, so it fails . We should be calling instead.

There is an ongoing that changes to utilize syscall to detect mount point by using . By using , bind mount will be properly detected fast, but it requires kernel version is v5.6 or later. So, we would be able to also utilize in . Then, the issue of bind mount can be resolved at least for kernel v5.6+, and we will be able to focus on how we solve this issue for old kernels.


Question: lbBackendPoolID := az.getBackendPoolID(lbName, lbBackendPoolName) lbIdleTimeout, err := getIdleTimeout(service) <del> if err != nil { </del> <ins> if wantLb && err != nil { </ins> return nil, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the Idle Timeout annotation was set to "3" which isn't a supported case, kube-controller-manager is stuck in deleting Azure LB. New service with the same name couldn't be created again even without the annotation. What you expected to happen: New service should be created successfully. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /kind bug
/assign Since I'm seeing this on some of our clusters
/priority important-soon
Thanks
- The problem here seems that when we want to call (when processing a delete event) it reconciles the loadbalancer which in turn tries to get the idleTimeout, resulting in an error. one of the strategies is to still construct our diff in case of a delete and exercise the code path at the end. This basically translate into ignoring the error when getting idleTimeout and excluding it from finding if the rule is dirty or not when we are . toughts ? Generally speaking, whenever we update a loadbalancer with unsupported values on annotations, ensuring load balancer will fail resulting in an un-cleaned LB, this is a cross-provider issue. /sig cloud-provider cc
cc
lgtm. the delete operation should continue to do cleanup job even some wrong options are provided from users.


Question: RemoveContainer(ctx context.Context, in *RemoveContainerRequest, opts ...grpc.CallOption) (*RemoveContainerResponse, error) // ListContainers lists all containers by filters. ListContainers(ctx context.Context, in *ListContainersRequest, opts ...grpc.CallOption) (*ListContainersResponse, error) <del> // ContainerStatus returns status of the container. </del> <ins> // ContainerStatus returns status of the container. If the container is not // present, returns an error. </ins> ContainerStatus(ctx context.Context, in *ContainerStatusRequest, opts ...grpc.CallOption) (*ContainerStatusResponse, error) // ExecSync runs a command in a container synchronously. ExecSync(ctx context.Context, in *ExecSyncRequest, opts ...grpc.CallOption) (*ExecSyncResponse, error)

The behavior when requested id doesn't exist is not defined clearly in CRI. Currently, we define that should return when requested image doesn't exist (), and kubelet is relying on this behavior now (). However, we haven't clearly defined the behavior of and . Currently, they return error when requested sandbox/container doesn't exist, and kubelet is also relying on this behavior. Not super urgent, just track the issue here. /cc
/assign
Added for clarifying the behavior.


Question: "//staging/src/k8s.io/client-go/tools/watch:go_default_library", "//test/e2e/framework:go_default_library", "//test/e2e/framework/events:go_default_library", <del> "//test/e2e/framework/kubelet:go_default_library", </del> "//test/e2e/framework/network:go_default_library", "//test/e2e/framework/node:go_default_library", "//test/e2e/framework/pod:go_default_library",

What happened: In reviewing a test for promotion to conformance I noticed the kubelet API was being accessed via the API server's node proxy endpoint (ref: ). This seemed off, so I reviewed . It doesn't explicitly call out kubelet API access as allowed or disallowed. . and agreed kubelet API access seems like a bad idea. I'm now surveying what we have that does_ assume kubelet API access. It's definitely used by parts of e2e framework and some Conformance tests. I'm trying to verify how critical its use is. It appears that much of it is related to debugging or verifying whether kubelets and the control plane agree on who is running what. What you expected to happen: Find a blanket policy that bans kubelet API access for Conformance tests, whether directly, or indirectly through the API server's node proxy endpoint, at any time. Alternatively, find a policy that says kubelet API access may be allowed, but only for debugging or failure purposes (must not be on the critical path to make the test pass) Alternatively, find a policy that says kubelet API access is allowed, but only indirectly through the API server, and evaluated against the same constraints we use for control plane access (GA, non-optional) If we want to say kubelet API access is banned, we can use this as a tracking issue to demote whichever tests rely on it, and come up with a plan to verify the same behavior without kubelet API access. /area conformance /sig architecture /priority important-soon /milestone v1.16 /assign
Look for anything that ends up calling calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test requires kubelet API access to pass may end up calling it if the test fails, via: Some Conformance tests may end up calling this if the test fails specifically Other areas this is used: only if the test fails required for a pre-stop hook test to pass required to make sure the readonly kubelet apis aren't exposed via api server proxy resource usage tracking tests require kubelet api access to pass tests call this in
Looking for doesn't turn up any additional conformance tests, and I don't feel compelled to link the two other tests that use it as they are unlikely to ever be eligible.

The examples pointed out by on slack, the kubelet's endpoints is required to be available for these to pass
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
are there further changes for this targeted at 1.16?
Any update on this?
/milestone v1.17
Bug Triage Release Team checking in. Code freeze for the 1.17 release is on November 14. Will this make it in? Thanks!
/assign
hi is this still targetting for 1.17 or can we move to 1.18? Code freeze is Nov 14th. thanks!
/milestone v1.18
So we've updated the documented policy. My suggestion to close this out is to drop the [Conformance] tag from the tests that currently require the kubelet API to pass, and open issues that they need to be rewritten. Ignore whether the API is needed for diagnostic info in failure modes. SchedulerPredicates [Serial] calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test Pods should be submitted and removed [NodeConformance] requires kubelet API access to pass
i'd support this ^^
Hello , Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March. Is this issue still intended for milestone 1.18?
Hello There's been no movement on this issue this year. Should this get pushed to the next milestone?
yes, please John T Skarbek <:
/milestone v1.19
/cc


Question: "net" "net/http" "strconv" <ins> "strings" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --As we update Kube-proxy to version 1.21, we observe a stream of following error entries within the Kube-proxy logs: Skipping invalid IP: ". As we do not see anything wrong with the EKS cluster itself, this message should not be reported as an error. an EKS cluster v1.21 and launched a managed nodegroup within this cluster. use the managed add-on or manage Kube-proxy yourself. Either way, the EKS provided kube-proxy image (kube-proxy:v1.21.2-eksbuild.2) is based on the upstream image and this error can be traced to the upstream code as I will be outlining in the next section. logs for the kube-proxy pod: kubectl logs -n kube-system <kube-proxy-pod-nameIn my replication, I could analyze this: These errors were recurring in the kube-proxy logs every 15 minutes which I concluded after watching the log stream for a 1+ hour window. To be able to make sense of the logs better, I increased verbosity of kubeproxy to 9: kubectl patch ds kube-proxy -n kube-system --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/command/1", "value":"--v=9"}]' In the logs, I could notice the following snippet recurring periodically (every 15 minutes): This hinted that a 'resync' was behind causing this errors being pushed every 15 minutes, however, before diving deeper into the root cause there, I searched for the origin of 'Skipping invalid IP: ' line and tried to reverse engineer back to the root cause for these errors. For this, I went through the '' in kube-proxy upstream github (release-1.21 branch). I started with the 'MapIPsByIPFamily' function which is where this error is handled and thrown [1]. This error is returned to it by a child function getIPFamilyFromIP [2]. The purpose of this function is to identify if the IP string passed is IPv6 or IPv4.

So, I backtracked a bit from the 'MapIPsByIPFamily' function and observed that it was referenced on line 186 by 'newBaseServiceInfo' in the file of kube-proxy [3]. Here I noticed that 'service.Spec.ExternalIPs' is passed as string to the 'MapIPsByIPFamily' function. Now, not all services have an ExternalIP defined in their spec. This is when an empty string '' is passed to the latter and hence the snippet : [klog.Errorf("Skipping invalid IP: %s", ip)] translates to 'Skipping invalid IP: ' in the actual error message. However, in the following error handling block: Rather than simply logging an error, there could be a check for this case of an empty string (as this is pretty common for clusters to have services without ExternalIP defined). If it is not possible to handle this error in this snippet for some reason, the verbosity level could be highered to atleast 4 so that the default kube-proxy image that runs at level 2 generally is not bombarded with a stream of recurring errors that are not actually errors. Can you please help fix this issue? [1] [2] [3] Kubernetes version (use ): 1.21 Cloud provider or hardware configuration: EKS with EC2 worker nodes OS (e.g: ): Amazon Linux 2 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network
/triage accepted nice report thanks /cc
Thanks. is your service defined with ?
I performed a 'kubectl get svc <svc_name-o yaml' and I could confirm that my services do not have an explicit .Spec.ExternalIPs = {""}. The output for one of them was as follows:
/assign
There was a similar problem with blank spaces too on the field , in this case the IPs were invalid because a blank space was appended to the IP If is not an ExternalIPs, then it has to be in the Status.LoadBalancer.Ingress.IP I think that if the function errors, we should return the error instead of logging inside, so the caller can handle it and raise a meaningful error, pointing the user to the specific service and field , what do you think?
iirc we have to ignore and log the error. Because there some cloud providers that sets this as " ".

And since there is no validation in place for that, we can not afford to add validation now (break the api). What i am looking into is limit the amount of times we log this error on proxy or remove it at all.
yeah, validation train has left the station some time ago :smile: I was trying to mean that the error should point to the service causing the issue, so users can act on it ...
Thanks for taking a look guys, really appreciate your help with this I could see that the fix has been planned as part of v1.23 milestone. Can you please clarify if this fix can/would be backported to previous Kubernetes versions and if there is a timeline for the same. This will help plan if we will continue to see such errors for some time now. Thanks again, good day !


Question: const tokenFilterName = "token-filter" <ins> // tokenFilter is a fake plugin that implements PreFilter and Filter. // `Token` simulates the allowed pods number a cluster can host. // If `EnablePreFilter` is set to false or `Token` is positive, PreFilter passes; otherwise returns Unschedulable // For each Filter() call, `Token` is decreased by one. When `Token` is positive, Filter passes; otherwise return // Unschedulable or UnschedulableAndUnresolvable (when `Unresolvable` is set to true) // AddPod()/RemovePod() adds/removes one token to the cluster to simulate the dryrun preemption </ins> type tokenFilter struct { <del> Tokens       int Unresolvable bool </del> <ins> Tokens          int Unresolvable    bool EnablePreFilter bool </ins> } // Name returns name of the plugin.

In k8s 1.27, scheduler introduced a new "short circuit" semantics in PreFilter to bypass a Pod quickly. A change introduced here is it relies on that plugin to be stored in , which makes sense as what it was explained . However, the info, although is populated upon each plugin's PreFilter invocation, is not injected into when some PreFilter returns unschedulable: So think about this workflow: pluginA returns Skip in its PreFilter pluginB return Unschedulable in its PreFilter note: pluginA is not included in due to aforementioned bug a Pod is claimed as unschedulable (failed by pluginB) during PostFilter, it's supposed to bypass pluginA's PreEnqueueExtension's AddPod/RemovePod, but it doesn't calling a "skipped" Plugin's AddPod/RemovePod during preemption, sometimes will abort the whole preemption as the logic tries to read info from cycleState, and some plugin's "skip" version doesn't fill cycleState properly, e.g., InterPodAffinity (skip immediately w/o write preFilterStateKey). Short circuiting using Skip is fine, but should injected properly at all cases. Described in the workflow above. It's firstly captured in an out-of-tree PostFilter plugin: , but can also happen in in-tree plugins. No response v1.27 <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind regression /sig scheduling
cc
Among in-tree plugins, 3 plugins employ non-nil and may return Skip in their PreFilter() implementation w/o writing to cycleState, while return error immediately if is missing in their Add/RemovePod() functions.

So technically, as long as a scheduling flow hit the following criteria, it would hit the bug (preemption doesn't function): plugin 1/2/3 return Skip in PreFilter arbitrary PreFilter plugin after the "skipped" plugin returns Unschedulable in its PreFilter
/cc
/cc
Cherry-pick PRs: 1.28: 1.27:


Question: local binaries binaries=($(kube::golang::binaries_from_targets "${targets[@]}")) <del> kube::log::status "Building go targets for ${platforms[@]} in parallel (output will appear in a burst when complete):" "${targets[@]}" </del> local platform <del> for platform in "${platforms[@]}"; do ( kube::golang::set_platform_envs "${platform}" kube::log::status "${platform}: Parallel go build started" </del> <ins> for platform in "${platforms[@]}"; do kube::golang::set_platform_envs "${platform}" kube::log::status "Building go targets for ${platform}:" "${targets[@]}" local -a statics=() local -a nonstatics=() for binary in "${binaries[@]}"; do if kube::golang::is_statically_linked_library "${binary}"; then kube::golang::exit_if_stdlib_not_installed; statics+=($binary) else nonstatics+=($binary) fi done if [[ -n ${use_go_build:-} ]]; then # Try and replicate the native binary placement of go install without # calling go install.  This means we have to iterate each binary. local output_path="${KUBE_GOPATH}/bin" if [[ $platform != $host_platform ]]; then output_path="${output_path}/${platform////_}" fi </ins> <del> local -a statics=() local -a nonstatics=() </del> for binary in "${binaries[@]}"; do <ins> local bin=$(basename "${binary}") if [[ ${GOOS} == "windows" ]]; then bin="${bin}.exe" fi </ins> if kube::golang::is_statically_linked_library "${binary}"; then kube::golang::exit_if_stdlib_not_installed; <del> statics+=($binary) </del> <ins> CGO_ENABLED=0 go build -installsuffix cgo -o "${output_path}/${bin}"  "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  "${binary}" </ins> else <del> nonstatics+=($binary) </del> <ins> go build -o "${output_path}/${bin}"  "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  "${binary}" </ins> fi done <del> if [[ -n ${use_go_build:-} ]]; then # Try and replicate the native binary placement of go install without # calling go install.  This means we have to iterate each binary. local output_path="${KUBE_GOPATH}/bin" if [[ $platform != $host_platform ]]; then output_path="${output_path}/${platform////_}" fi for binary in "${binaries[@]}"; do local bin=$(basename "${binary}") if [[ ${GOOS} == "windows" ]]; then bin="${bin}.exe" fi if kube::golang::is_statically_linked_library "${binary}"; then kube::golang::exit_if_stdlib_not_installed; CGO_ENABLED=0 go build -installsuffix cgo -o "${output_path}/${bin}"  "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  "${binary}" else go build -o "${output_path}/${bin}"  "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  "${binary}" fi done else </del> <ins> else </ins> # Use go install. if [[ "${#nonstatics[@]}" != 0 ]]; then go install "${goflags[@]:+${goflags[@]}}"  <del> -ldflags "${version_ldflags}"  "${nonstatics[@]:+${nonstatics[@]}}" </del> <ins> -ldflags "${version_ldflags}"  "${nonstatics[@]:+${nonstatics[@]}}" </ins> fi if [[ "${#statics[@]}" != 0 ]]; then CGO_ENABLED=0 go install -installsuffix cgo "${goflags[@]:+${goflags[@]}}"  -ldflags "${version_ldflags}"  <del> "${statics[@]:+${statics[@]}}" </del> <ins> "${statics[@]:+${statics[@]}}" </ins> fi <del> fi kube::log::status "${platform}: Parallel go build finished" ) &> "/tmp//${platform////_}.build" & done local fails=0 for job in $(jobs -p); do wait ${job} || let "fails+=1" done for platform in "${platforms[@]}"; do cat "/tmp//${platform////_}.build" </del> <ins> fi </ins> done <del> exit ${fails} </del> ) }

After 6 minutes or so of using 50% my 4-core MBP CPU and chewing up a ton of memory, make release gives up with a timeout.
This is on my master branch (no local changes) after rebasing to head this evening. It was working fine earlier today so seems like the likely culprit. /cc
It's certainly my changes. I'm on a low function device, feel free to revert. This is only an issue for low memory systems, I suspect. :/ On Apr 1, 2015 10:24 PM, "Robert Bailey" wrote:
(This is functioning fine on Jenkins and my desktop, FWIW.) On Apr 1, 2015 10:36 PM, "Zachary Loafman" wrote:


Question: } podHandleHookRequest := e2epod.NewAgnhostPod("", "pod-handle-http-request", nil, nil, ports, "netexec") ginkgo.BeforeEach(func() { <ins> node, err := e2enode.GetRandomReadySchedulableNode(f.ClientSet) framework.ExpectNoError(err) targetNode = node.Name nodeSelection := e2epod.NodeSelection{} e2epod.SetAffinity(&nodeSelection, targetNode) e2epod.SetNodeSelection(&podHandleHookRequest.Spec, nodeSelection) </ins> podClient = f.PodClient() ginkgo.By("create the container to handle the HTTPGet hook request.") newPod := podClient.CreateSync(podHandleHookRequest)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --k8s-e2e-ltsc2019-containerd-flannel-sdnoverlay-master [] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestart http hook properly [NodeConformance] [Conformance] [] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance] Note: the tests have been excluded until they are fixed. In the case of multinode clusters, the http server pod and the test cluster can spawn on different nodes, which can be problematic for poststart / prestop hooks, as they are executed by the kubelet itself, and the cross-node lifecycle hook might fail (according to the [Kubernetes network model], it is not mandatory for kubelet to be able to access pods on a different node). However, that is not always the case, different Networking setups allow cross-node communication between kubelet and pods, which means that the above tests are passing properly (for example: ), which begs the question if this behaviour should be universal or not, especially since the tests are marked as , which would imply that it should work in all cases. [Kubernetes network model]: links to appreciated links to specific failures in spyglass appreciated Slack: /sig testing /sig networking /sig windows <!-- Please see the deflaking doc () for more guidance! --
/sig network /sig node /area conformance /assign /priority critical-urgent /cc This is important since affects conformance and defines the networking behaviour for the poststart/ prestop hooks
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig node /sig conformance /assign /priority critical-urgent /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i'd personally let folks have first dibs on what we need to do here. Though from the name of the test itself, it seems like the test is NOT a sig-network test :)
Indeed, it does not have any tag in its name, but the tests rely on networking and on how CNIs work.

As I've stated above, it's not a problem for flannel sdnbridge, but it is for flannel sdnoverlay; other CNIs might have a similar issue too.
because the network is not a thing :) let´s tag the sig owner then /sig node I think that the key here is to decide if kubelet should be able to access pod in other nodes, and based on the link Claudiu is part of the Kubernetes networking model. I saw users that doesn't want kubelet to access even the pods on same host, so I think that is legit to say kubelet MAY reach pods in other nodes but not MUST ... but I'm not familiar with the to fully understand the context. If kubelet "may" reach pods in the other nodes, the test must only test lifecycle hooks for pods in the same node as the kubelet under test, right?
This is correct. The model also says: "For those platforms that support Pods running in the host network (e.g. Linux): pods in the host network of a node can communicate with all pods on all nodes without NAT" I believe that Windows host network is not supported, right?
Right, and there are specific tests that checks that exact scenario you've mentioned: However, the flaky tests I've mentioned in the issue require kubelet to be able to dial-in into a webserver pod for a different pod's poststart / prestop HTTP lifecycle hooks. There are couple of things regarding this: of all, in order to ensure kubelet would be able to run the HTTP lifecycle hooks, we'd have to make sure both pods are on the same node, as that is a supported scenario, according to the Kubernetes Network Model. I have sent a PR for this above. Alternatively, a Cluster IP could probably work as well, but IMO, ensuring that both pods end up on the same node is simpler and easier to clean up. , it's interesting that those HTTP lifecycle hook tests work on different networking setups, without any pods having HostNetwork enabled, even though you've mentioned they should (e.g.: Linux / Windows with flannel l2bridge, and even Linux with flannel overlay). Now, my question is: is this behaviour a feature, or a bug, especially since the tests are marked as `?

oh, I missed that part, so in Linux kubelet must be able to communicate with pods on all nodes (becuase it is running in the hostnetwork) If that is correct, it seems that one solution is to keep the test for Linux and tag it as LinuxOnly, and a new test for WindowsOnly that only spawn the pods under test in the same node EDIT Discussed in sig-net meeting , no clear answer... can be kubelet considered as a hostnetwork pod?
For the purposes of testing, I don't mind forcing the pods onto the same node., but let's document the heck out of that at the code-site?
correct but it will be supported once HostProcess containers are merged in 1.22: Only in Overlay networks (ie Flannel where these tests are flaking) does the pod have to be on the same node. It doesn't require host network mode and these tests are . I would be more inclined to disable this test for Windows Overlay test setups and have a sig-windows test for the overlay functionality. I would also question if the lifecycle hooks only running on the same node is useful and if it should just be documented that it doesn't work in the Windows Overlay setup.
is looking at test PRs from SIG Node. I will let sig-net triage this issue
Next steps on this?
/triage accepted
We still haven't merged this: I also had this, but it has been decided to be blocked:
requires conformance approval, at this point I think that has to be postponed to 1.23 right, since that behavior is not required, we shouldn't test it, because is legit to fail and having a test for something that can fail can create confusion :smile:


Question: import ( apiequality "k8s.io/apimachinery/pkg/api/equality" apimachineryvalidation "k8s.io/apimachinery/pkg/api/validation" <ins> pathvalidation "k8s.io/apimachinery/pkg/api/validation/path" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" <ins> utilvalidation "k8s.io/apimachinery/pkg/util/validation" </ins> "k8s.io/apimachinery/pkg/util/validation/field" corevalidation "k8s.io/kubernetes/pkg/apis/core/validation" "k8s.io/kubernetes/pkg/apis/resource"

Following YAML file is accepted without error. Notice the field, it is an instead of , containing character. This can lead to an issue when resource driver generates structured parameters' in-tree object with being just without version. The DRA scheduler plugin will try to match the parameters object based on the reference here: In latest master the code appears to use an indexer, and the issue might be hidden, if applicable at all. I did not check the reproducibility in master yet. However, main question is whether the should allow value with character in general. An error is expected from the apiserver upon object creation request, reporting that APIGroup containes APIVersion. Maybe. Have any resource driver that supports structured parameters, , for instance, also puts (gpucrd.APIGroup) value into . v1.30.1 <details</details<details</details<details</details<details</details<detailsscheduler's dynamicresources plugin </details
/sig node /triage accepted
/assign


Question: <ins> apiVersion: v1 kind: PersistentVolumeClaim metadata: name: {{.FEDERATION_APISERVER_DEPLOYMENT_NAME}}-etcd-claim annotations: volume.alpha.kubernetes.io/storage-class: "yes" namespace: {{.FEDERATION_NAMESPACE}} labels: app: federated-cluster spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi </ins>

As mentioned in , the federation etcd will lose its state any time the pod dies, unless we modify to add a PVC. We'll also need to create a PV and modify the turnup script to create the backing volume in the cloud provider.
FYI I believe this is a showstopper.
OK. implied I could use to get this provisioned on-demand. AFAICT, , and since it makes our life easier I am trying to get this to work in my test cluster, but so far no joy. Any obvious mistake?
I needed to not specify a .
whew! , Matt Liggett wrote:


Question: "type": "Testing"}, }, Spec: v1.PodSpec{ <ins> TerminationGracePeriodSeconds: &zero, </ins> Containers: []v1.Container{{ Image: imageutils.GetE2EImage(imageutils.Agnhost), Name:  "token-test",

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce (5% of runs) pull-kubernetes-e2e-kind-ipv6 pull-kubernetes-e2e-gce-ubuntu-containerd Which test(s) are flaking: This is the flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
merged 2 weeks ago in
/assign
potential mitigation in
To track down the test flaking there was some great feedback in by and was created as a potential fix based on that feedback /cc


Question: // resetFieldsStatusDefault conflicts with statusDefault const resetFieldsStatusDefault = `{"status": {"conditions": [{"type": "MyStatus", "status":"False"}]}}` <del> var resetFieldsSkippedResources = map[string]struct{}{} </del> <ins> var resetFieldsSkippedResources = map[string]struct{}{ // TODO: flowschemas is flaking, // possible bug in the flowschemas controller. "flowschemas": {}, } </ins> // noConflicts is the set of reources for which // a conflict cannot occur.

A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted


Question: // Find all sibling pods in the service and post to their /write handler. func contactOthers(state *State) { <ins> const waitTimeout = 2 * time.Minute </ins> defer state.doneContactingPeers() client, err := client.NewInCluster() if err != nil {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : The documentation suggests that "The default pull policy is IfNotPresent" This is not the case when imagePullPolicy is defined under the spec section in a Pod definition without a specific value. Example: This will result in a Pod having imagePullPolicy as Always. Why is this needed: This is needed for clarification of the default behaviour. It is necessary for the appropriate handling of helm charts, in cases that the imagePullPolicy is empty.
/sig docs
The imagePullPolicy default value is .
In this paragraph you can see that In your reference it points out the default action for the kubelet, not the behaviour when imagePullPolicy is declared. Please do have in mind that the imagePullPolicy option for Never is mentioned only in this paragraph in the document.
If an image tag of "latest" is used, the imagePullPolicy defaults to to "Always": If no tag and no digest are specified on the image, an image tag of "latest" is used: The behavior when no image digest or tag is specified should be documented in the API docs for the and fields:
I am interested on helping this issue, but I am new to kubernete repo, it could take me sometime
I found the following which is well documented in "Best Practices for Configuration" It would be expected to include more information on the options in the main link under
I'd like to work on this one.
Very welcome, but yesterday also mentioned that he is following up on this issue, you can communicate with him before starting

cc
This has failed The two failures are different, and both seemingly totally legitimate. The first one is simply a pod never entering running state within the 5 minute timeout. Expected error: <*errors.errorString | 0xc208475150: { s: "gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s", } gave up waiting for pod 'nettest-dtwk3' to be 'running' after 5m0s not to have occurred The second is not being reachable: Dec 7 01:22:47.275: Unable to connect/talk to the internet: Get dial tcp 74.125.202.147:80: i/o timeout The latter could be a result of a missing MASQ rule on the node, or GCP egress problems. I don't think that this is worth trying to debug further in the context of this test for intra-pod communication. I'll check more generally whether we have a current intermittent problems with respect to pods entering running state, or reaching the internet.
I think this test may be causing builds to time out, too. See comments from me in .
For the future, I actually don't think we need the test at all in the networking test. Maybe even move it do a different test altogether.
This is clearly a bug. I'll send a PR shortly: Regarding the ping to , why did you put in in the first place?
hmmm, not sure what you mean by "in the first place" ? I did not write the test...
sorry, I thought that you'd written that part of the test. I will find out who wrote that, figure out why they put it there in the first place, and then either remove it, or rewrite it to work correctly. Clearly that part of the test needs at least some basic retry to work consistently.
ok great, no problem just wasnt sure what you meant. SGTM. Id be okay w/ deleting the ping entirely regardless of who wrote it, its a very old snippet which doesnt really add much to the overall test suite (and in fact is questionable wether it should even be an assertion at all )
somebody put it there for some reason. Better to understand why before just deleting it.

i looked, its actually rob who i work with :) i will ask him in the morning unless you find him first :) (rrati)
Regarding my alleged "clear bug" above, I was completely wrong. There's nothing wring with that code. And unfortunately, when this test goes into a presumed endless loop (see comment above) it does not produce any output (as the output is cached by Ginkgo until the test completes). So it's very difficult to debug. But in summary I agree with this test is probably never completing, and causing tests to time out. Someone will need to isolate this test in a environment where it produces output and can be debugged. I'll keep trying, but need to head home soon, so if anyone wants to pick it up overnight, go ahead.
Smoking gun:
hmm - I'm just wondering why only this particular test is hanging forever (if we agree that's the case). I don' think that some requests are hanging forever in this test, but not in any other test...
Closing in favor of which should fix this. Or at least some way to set client timeout, which and are still figuring out.
It's happening again.
Talked to offline. I am assigning this one to you for triage. Let me know if you need any help.
AFAICT the last failure was just the pod nettest-7h5jj not coming up
Looks like this should be resolved by your pr: too. :-)
Maybe. Though the webserver only deadlocks in the failure case, so the test is probably still flaky...
Looks like this test is now explicitly flaking by failing, as I expected.
Based on the , it took 15 minutes to pull the image. closing this one as dup of
New Failure:
This looks like a race between populating the endpoints list and the /status, /read methods of the service exposed through the master proxy. It looks like the webserver pod is just trying to confirm that it can reach all peers. We can do this synchronously after having observed all expected endpoints in the test (or re-write the test to use the same pattern we do in services e2e).
The test looks really racy to me. I think the following 15 retries prevents it to fail more often, but not completely.

I will submit a pr to enfore webservers to wait till alll endpoints are up before they start to contact each other.
New failure
I am making this P0
It took more than 4 minutes for a nettest pod to come up. The nettest container is waiting for all endpoints to be ready for 2 minutes plus 15 attempts (5 sec interval) to contact its peers. This happens in less than 4 minutes. Hence the first 2 nettest pods finished contacting peers and declared failure.
The node you want is: I see: that's too much silence ...
From the It looks like kubelet is slow to start the container. Due to the lower logging level in the gke, it's hard to guess what could've slowed down the goroutine...


Question: gsutil mb "${staging_bucket}" fi <del> local -r staging_path="${staging_bucket}/devel${KUBE_GCS_STAGING_PATH_SUFFIX}" </del> <ins> local -r staging_path="${staging_bucket}/${INSTANCE_PREFIX}-devel" </ins> SERVER_BINARY_TAR_HASH=$(sha1sum-file "${SERVER_BINARY_TAR}") SALT_TAR_HASH=$(sha1sum-file "${SALT_TAR}")

We seem to have an occasional consistency issue between the client and server versions on our Jenkins GCE e2e project. Build 6295 failed this morning because the client version was newer than the server version: The server version matches the build was was immediately running in 6294, which was aborted early. I wonder if (somehow) the cleanup didn't work properly? (Which is a bit worrying if so.) Here's the version from 6294: A short while later, build 6297 failed, this time with the client version older than the server version: This one makes a little bit more sense: there was a new build published between downloading the client/test tarballs and setting up the cluster. I'm guessing that somewhere in the cluster setup scripts the latest CI release file was read again, rather than using the version read for the client/test tarballs. (I haven't yet worked out where exactly it's read, but it should be an easier fix.) cc
A side issue: we should probably figure out why the build git client is being marked dirty. Since we're checking out the head version from GitHub, it should ideally be a clean client.
The -dirty bit is now fixed by .
This is caused by different Jenkins jobs uploading to the same GCS bucket and clobbering each other. In we see: Over in
This also explains the issues seen in ; the job is also running in the same project.
Ah, good sleuthing. We avoided this issue in the PR builds by setting (used ), but maybe we should always be setting to include the Jenkins job name (and optionally executor/node number).
The original idea was to run every e2e job in it's own project, to avoid these sorts of name clashes. But we seem to have regressed and now have many jobs sharing the same projects. Is project-per-job not a better approach? It also helps with resource leaks and the likes. Just a thought.
I'd argue this is actually uncovering a (dev) deployment bug; I should be able to deploy different versions of Kubernetes to the same project. I know you and I disagree (and probably always will) on the strategy here.

:smile:
fwiw I think I agree with that if project-per-job avoids this issue and helps identify resource leaks, it's a pragmatic near-term improvement in the state of the world that would result in more benefit than the cost of changing all the everythings to support multi-job-in-project. Re: devs deploying different versions of k8s to the same project, P3 feature request?
I was just about to suggest the same thing. I will file a separate issue, and fix this in the short term an easier way. is out to make it easier to isolate a run to a project.
Also it seems like a reasonable thing (higher than P2) to want to deploy multiple clusters to the same project safely in prod, not just dev. Disagree?
Multiple clusters in prod using well-defined build versions such as v1.1.4 = P0. Multiple clusters in prod using our cooked-up test-building system = non-goal?
SGTM; this is prod code, though, not test code; it's used every time we call , regardless of whether it's a CI build or released version.
FYI: kubernetes-e2e-gce-reboot/ has one reboot test failed due to kube-proxy image change. But that build doesn't include pr yet.
looks like it. That run used version , but the first build with was .


Question: assert.NotNil(t, createdNamespace) assert.Equal(t, ns1.Name, createdNamespace.Name) <ins> // Wait for the secret to appear in the informer store err := WaitForStoreUpdate( namespaceController.namespaceFederatedInformer.GetTargetStore(), cluster1.Name, ns1.Name, wait.ForeverTestTimeout) assert.Nil(t, err, "namespace should have appeared in the informer store") </ins> // Test update federated namespace. ns1.Annotations = map[string]string{ "A": "B",

https://k8s- Failed: TestNamespaceController {namespace} Previous issues for this test:
GetObjectFromChan() is timing out (20s) and returning nil causing the panic. Happens in TestSecretController (and probably TestIngressController) controller as well. https://k8s-
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
I have a fix for this in
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
https://k8s- Failed: TestNamespaceController {namespace}
/cc OWNERS of :
I think this is also related to Issue .
fix is pending


Question: import ( "fmt" <ins> "strings" </ins> "time" "k8s.io/apimachinery/pkg/api/errors"

In federation tests, if we observe the build logs. we see cleaning up LoadBalancer from cloud provider seems not working. It has 2 effects: delays the tests by waiting to delete but always returns error. so we waste like 5 minutes. maybe we are leaking the LoadBalancer in the project.
we can see these errors in every build. but just for reference here are some build logs https://k8s-
cc
All the errors in logs are . It doesn't necessarily indicate we are leaking resource, just now we must have clusterID initialized before calling EnsureLoadBalancerDeleted(). Would be better to not call this function directly in e2e test :)
Federation cleanup should use the same implementation as non-federated e2e test:
function looks like only specific to GCE. in future we would run the e2e tests in AWS too. it would be better to use generic functions if any.
using is working for us. here is the test logs https://k8s-
EnsureLoadBalancerResourcesDeleted() actually cleans up nothing, it only lists forwarding rules and probably logging :)


Question: // Metrics for total request latency start := time.Now() defer func() { <del> metrics.RequestLatency.Observe(ctx, r.verb, *r.URL(), time.Since(start)) </del> <ins> metrics.RequestLatency.Observe(ctx, r.verb, r.finalURLTemplate(), time.Since(start)) </ins> }() if r.err != nil {

Hello, we have been using client-go to measure the latency for all client-go traffic from Cilium to kube-apiserver. We have that since v1.24.0 the URL path is no longer aggregated in a way that this cardinality would no longer happen as presented in the following image: <img width="330" alt="Screen Shot 2022-08-08 at 10 18 24 AM" src="https://user-"It seems this functionality was removed as a part of as it seems that PR only took into account that users of the would only be using it to measure latency to a particular host and not for a particular URL path as we are doing in . Would it be reasonable to revert commit in order to fix this regression? Thanks cc
/sig api-machinery /sig instrumentation
But reverting will bring back the cardinality problem, will not? The latency of each URL path is available in the apiserver metrics ...
We have never any cardinality problem with the previous code since we were not using the "host" field which is what I assume the original issue about cardinality was trying to fix. We only used the "path" of an URL, which was aggregated in a way that would not be facing any cardinality, like the figure attached in this GH description. Now, starting with = 1.24.0, each and is actually a different value which does cause the cardinality problem to appear.
ah, I see the disconnection, I wasn't understanding the issue sorry. The Interface uses the url as parameter to keep backwards compatibility, but the method that gathers the metrics only uses the host to avoid the cardinality problem. You have to update the method that gathers the metrics to only use the host part of the url
Filed
should fix the code in kubernetes ? however, that will break consumer of the Interface that still wants to process the path, but we know using the path will cause issues with cardinality :shrug:
I would be fine with adding a new label to mitigate that issue since originally the cardinality explosion that was reported was caused by the query string and not the path: When trimming down the whole URL to only the host we even had a discussion to keep the path, but we decided not to since we didn't have any use case for this data.

If you have a use case where you need the path in order to get more granularity on the latency of your network then I would be fine with adding the label to Kubernetes, but another path forward could be to also remove the path information from cilium if deemed unnecessary. In the most common scenarios, you will only need the host to get the network latency based on different proxy routes.
yeah we can't really do much about it, it is either we leak the unsanitized paths or we prevent the users from using the path completely. Since we want to not break the interface and keep the URL parameter, we might just have to keep sanitizing the paths for our consumers even though we are not using the path anymore.
IIUIC the RESTClient is used in the dynamic client-go too, for these users with 1900 CRDs, the path is really going to explode for them
Then it seems that the best option to not break users would be to revert , but at the same time I don't really like the idea of spending CPU time sanitizing something that will never be used in the RESTClient
A suggestion would be to create a new interface similar to the that would have the following method: Thus, those users with 1900 CRDs would use the instead of the .
after digging a little bit the underlying issue it looks that the problem was initially caused by a kubelet bug as pointed out . The dependents of client-go don't have access to all the fields of the struct which uses heavily to generate the aggregated URL.
well, it is clear now that current status is wrong, we pass the whole URL to the interface and rely on the to use only the Host part of the URL :smile: I'm fine with whatever you decide, to restore or just completely strip the Path Sorry for the unfortunate consequence, at that time I didn't consider that the Interface/Method drift and thought that function was orphan
/triage accepted /assign


Question: clientset "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" "k8s.io/kubernetes/pkg/runtime" deploymentutil "k8s.io/kubernetes/pkg/util/deployment" <del> "k8s.io/kubernetes/pkg/util/errors" </del> <ins> sliceutil "k8s.io/kubernetes/pkg/util/slice" </ins> ) const (

Ref and should sort revisions correctly (10 should be after 9, not 1).
I can help with this bug. Looks like trivial fix in PrintRolloutHistory
Thanks
are you still taking care of this one ? I am searching for some easy to do PR to just get familiar with the process and put my first PR in.
I created pr some time ago to close this issue
great thanks :-)
you can look for help-wanted issues or check and
thanks


Question: func newAdmissionMetrics() *AdmissionMetrics { // Admission metrics for a step of the admission flow. The entire admission flow is broken down into a series of steps // Each step is identified by a distinct type label value. <del> step := newMetricSet("step", []string{"type", "operation", "rejected"}, "Admission sub-step %s, broken out for each operation and API resource and step type (validate or admit).", true) </del> <ins> // Use buckets ranging from 5 ms to 2.5 seconds. step := &metricSet{ latencies: metrics.NewHistogramVec( &metrics.HistogramOpts{ Namespace:      namespace, Subsystem:      subsystem, Name:           "step_admission_duration_seconds", Help:           "Admission sub-step latency histogram in seconds, broken out for each operation and API resource and step type (validate or admit).", Buckets:        []float64{0.005, 0.025, 0.1, 0.5, 1.0, 2.5}, StabilityLevel: metrics.STABLE, }, []string{"type", "operation", "rejected"}, ), latenciesSummary: metrics.NewSummaryVec( &metrics.SummaryOpts{ Namespace:      namespace, Subsystem:      subsystem, Name:           "step_admission_duration_seconds_summary", Help:           "Admission sub-step latency summary in seconds, broken out for each operation and API resource and step type (validate or admit).", MaxAge:         latencySummaryMaxAge, StabilityLevel: metrics.ALPHA, }, []string{"type", "operation", "rejected"}, ), } </ins> // Built-in admission controller metrics. Each admission controller is identified by name. <del> controller := newMetricSet("controller", []string{"name", "type", "operation", "rejected"}, "Admission controller %s, identified by name and broken out for each operation and API resource and type (validate or admit).", false) </del> <ins> // Use buckets ranging from 5 ms to 2.5 seconds. controller := &metricSet{ latencies: metrics.NewHistogramVec( &metrics.HistogramOpts{ Namespace:      namespace, Subsystem:      subsystem, Name:           "controller_admission_duration_seconds", Help:           "Admission controller latency histogram in seconds, identified by name and broken out for each operation and API resource and type (validate or admit).", Buckets:        []float64{0.005, 0.025, 0.1, 0.5, 1.0, 2.5}, StabilityLevel: metrics.STABLE, }, []string{"name", "type", "operation", "rejected"}, ), latenciesSummary: nil, } </ins> // Admission webhook metrics. Each webhook is identified by name. <del> webhook := newMetricSet("webhook", []string{"name", "type", "operation", "rejected"}, "Admission webhook %s, identified by name and broken out for each operation and API resource and type (validate or admit).", false) </del> <ins> // Use buckets ranging from 5 ms to 2.5 seconds (admission webhooks timeout at 30 seconds by default). webhook := &metricSet{ latencies: metrics.NewHistogramVec( &metrics.HistogramOpts{ Namespace:      namespace, Subsystem:      subsystem, Name:           "webhook_admission_duration_seconds", Help:           "Admission webhook latency histogram in seconds, identified by name and broken out for each operation and API resource and type (validate or admit).", Buckets:        []float64{0.005, 0.025, 0.1, 0.5, 1.0, 2.5}, StabilityLevel: metrics.STABLE, }, []string{"name", "type", "operation", "rejected"}, ), latenciesSummary: nil, } </ins> webhookRejection := metrics.NewCounterVec( &metrics.CounterOpts{

I would like to propose the graduation of the following apiserver metrics from alpha to stable: With the exception of the metric these metrics have been stable for 2 years. The metric has been stable for 12 months.
/sig api-machinery /sig instrumentation
/assign /triage accepted
I'm okay with promoting all of these except I am hesitant about since it is relatively new. Can we omit that one?
I agree with the metric was patched a year ago as part of , so it would be best to wait a bit more until the majority of Kubernetes users catch up to the patched version and start using the new version of the metric in production.
A couple of renaming suggestions, since it is important to get it right before graduation: apiserverresponsesizes =apiserverresponsesizebytes aggregatorunavailableapiservice =aggregatorunavailableapiservices One concern regarding . It was very recently changed but the change was later reverted by In the end, the metric wasn't really changed, but it might raise the concern that some users may still need additional information from this metric so I am on the fence with this one. Side implementation note that we will need to keep in mind if we graduate is that the graduation should only affect the webhook admission metric and not the other, so we will need to pass the stability level to
Thank you and for the feedback. To clarify then, are the decisions to not graduate and final?
I'm actually okay with , the namespace addition was recent but it was a bad idea and was reverted. It has otherwise been in use for quite some time, so I'd be comfortable with promoting it.
I'm willing to make a PR for this one
+1 it wasn't a hard no from my side, just wanted to raise some concerns in case anyone had an objection.
/assign /unassign
Thank you for the feedback, and To summarize, we are in agreement to graduate all but the metric. Please correct me if I'm wrong.
/cc
since all the three metrics that are created with this function are graduating to stable, do you think I should still pass the to the function or should I just set the metrics created by this functions as stable by default? update: Seems like is designed to parse whatever's in front of . So putting a variable there (that is passed to the function) will cause the script to fail. Please correct me if I'm wrong.

update: Even setting the metrics to be stable by default will cause the script to fail and you'll get an error saying for the following line: It appears that the name of the stable metrics should be given to that object as a string and using the output of a function ( in this case) will cause the script to fail.
The PR is ready to be reviewed:


Question: "k8s.io/kubernetes/pkg/kubelet/cm/cpumanager/topology" "k8s.io/kubernetes/pkg/kubelet/cm/cpuset" "k8s.io/kubernetes/pkg/kubelet/cm/topologymanager" <ins> "k8s.io/kubernetes/pkg/kubelet/config" </ins> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" "k8s.io/kubernetes/pkg/kubelet/status" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: if err != nil { return err } <ins> if len(infos) == 0 { return errors.New("edit cancelled, no objects found.") } </ins> return editFn(infos) case ApplyEditMode: infos, err := o.OriginalResult.Infos()

Try running and adding an ingress object, or an empty list, then exit the editor – it will panic like this: I've also tried , and for the sake of experiment tried adding a Pod, then it failed rather gracefully with this message: But when I tried to add a DaemonSet, it paniced. I am not sure how we should fix, perhaps editing of lists isn't something a normal user is expected to do... Let's discuss.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cli
So I wanted to test how it would behave if I take an item out of my list, so I've create and ingress object and tried . To my surprise I've found only one ingress object without a list. When I a second object, gave me a list to edit. So I must say that flattening of the list in case there is only one object was a bit surprising. Now that I have a list of two items, I've decided to try deleting one by removing it from the list, and it turns that's not possible.
It feels like the problem is complicated, and I'd rather see disallowed unless users passed a name of a specific resource. If we belive some folks rely on to work when there is only one resource to edit, maybe we can keep that... If we cannot disallow this, I'd rather see it fixed in a way where removing items results in deletion and editing a single item list is actually about editing a list. At the very least, of course, we should fix the panic, but I still want to hear what people think about this before I open a PR (presumably fixing that panic should be trivial).
I'm working on this. Already submitted a PR in . PTAL.
I think at the very beginning, at least v1.5.0 the logic is already like this, I'm not sure why this bothering you I use this lot of time, I'd prefer keep it.

also cc
I don't think edit should open with an empty list. should open a list if there are multiple items, a single item if there is only one, and error if there are none of that type. should open a single item if it is found, and error if it is not found
that sounds good to me. I was just a bit surprised, that's all.


Question: pod_memory_working_set_bytes{namespace="namespace_a",pod="pod_a"} 1000 1624396278302 `, }, <ins> { name: "nil pod metrics", summary: &statsapi.Summary{ Pods: []statsapi.PodStats{ { PodRef: statsapi.PodReference{ Name:      "pod_a", Namespace: "namespace_a", }, CPU: &statsapi.CPUStats{ Time:                 testTime, UsageCoreNanoSeconds: nil, }, Memory: &statsapi.MemoryStats{ Time:            testTime, WorkingSetBytes: nil, }, }, }, }, summaryErr: nil, expectedMetrics: ` # HELP scrape_error [ALPHA] 1 if there was an error while getting container metrics, 0 otherwise # TYPE scrape_error gauge scrape_error 0 `, }, </ins> } for _, test := range tests {

kubelet panic when accessing metrics/resource endpoint kubelet log kubelet works fine access metrics/resource endpoint of kubelet No response <details</details<details</details<details</details<details</details<detailsINFO[2022-07-14 05:36:21.474173892Z] Starting CRI-O, version: 1.23.0, git: (dirty) Version: 1.23.0 GitCommit: GitTreeState: dirty BuildDate: 1980-01-01T00:00:00Z GoVersion: go1.17.4 Compiler: gc Platform: linux/amd64 Linkmode: static BuildTags: apparmor, excludegraphdriverdevicemapper, seccomp, selinux SeccompEnabled: true AppArmorEnabled: true </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig node


Question: } } <ins> errChan := make(chan error, len(nodes)) var wg sync.WaitGroup </ins> // we need the max one to keep the same cpu/mem use rate ratio = math.Max(maxCPUFraction, maxMemFraction) for _, node := range nodes {

From After , Multi-AZ Clusters tests in 1.19 and 1.20 started timing out as createBalancedPodForNodes is extremely inefficient in scale: for each node (out of 5k), it lists all pods in a cluster serially (each list takes ~5s), which translates to ~7 hours. Optimizing this function by listing pods once and then in memory by Spec.NodeName /kind bug /priority critical-urgent /sig scheduling
/triage accepted
/assign
/reopen Looks that ComputeCPUMemFraction took only 12 minutes, but it stuck later, on It is scheduling one pod per each node: Which is waiting 5s to be ready: 5k nodes * 5s = ~7h
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Can you do the pod creations in parallel?
Yes, good idea. I'll update to do that


Question: dc.rsListerSynced = alwaysReady dc.dListerSynced = alwaysReady dc.ssListerSynced = alwaysReady <del> dc.recorder = record.NewFakeRecorder(10) </del> <ins> dc.recorder = record.NewFakeRecorder(100) </ins> informerFactory.Start(ctx.Done()) informerFactory.WaitForCacheSync(ctx.Done())

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // if there are any problems. dockerRoot := "/var/lib/docker" <del> // cgroup driver is only detectable in docker 1.12+ </del> <ins> // cgroup driver is only detectable in docker 1.11+ </ins> // when the execution driver is not detectable, we provide the cgroupfs form. // if your docker engine is configured to use the systemd cgroup driver, and you // want to use pod level cgroups, you must be on docker 1.12+ to ensure cgroup-parent

Kubernetes version (use ): v1.5.0-alpha.1.2189+-dirty docker: 1.12.3 Environment: fedora24 docker-engine-1.12.3-1.fc24.x86_64 What happened: if docker Cgroup Driver=cgroupfs, and set --cgroup-driver=systemd in kubelet, then enable --experimental-cgroups-per-qos=true; The container cgroup some is not correct. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know: 1) docker Cgroup Driver=cgroupfs and --cgroup-driver=cgroupfs in kubelet works well 2) docker Cgroup Driver=systemd and --cgroup-driver=systemd in kubelet works well
help check, thanks.
In Kubernetes 1.6, we can work to detect this case and return an error. IMHO its a configuration error to have docker running with a cgroupfs driver paired with the kubelet running the systemd cgroup driver. /cc
We can also document the compatibility matrix for now.