Question: restclient "k8s.io/client-go/rest" kubeversion "k8s.io/component-base/version" "k8s.io/kubernetes/pkg/api/legacyscheme" <del> "k8s.io/kubernetes/pkg/apis/batch" </del> flowcontrolv1beta2 "k8s.io/kubernetes/pkg/apis/flowcontrol/v1beta2" <del> "k8s.io/kubernetes/pkg/apis/networking" apisstorage "k8s.io/kubernetes/pkg/apis/storage" </del> "k8s.io/kubernetes/pkg/controlplane/reconcilers" "k8s.io/kubernetes/pkg/controlplane/storageversionhashdata" <ins> "k8s.io/kubernetes/pkg/kubeapiserver" </ins> kubeletclient "k8s.io/kubernetes/pkg/kubelet/client" certificatesrest "k8s.io/kubernetes/pkg/registry/certificates/rest" corerest "k8s.io/kubernetes/pkg/registry/core/rest"

Storage version defaults to the preferred version for a given group, but can be overridden when constructing API server storage. This is done here: Reasons the preferred version might be overridden: A given resource/kind does not yet exist in the preferred version (like when introducing a new alpha type in an existing API group with more stable versions) A given resource/kind is continuing to store in etcd in an older version for compatibility with n-1 API servers The etcd integration test verifies that we're actually storing the correct versions in etcd, but the computation of the StorageVersionHash is not honoring the overridden version. on master, this code reports v1 CSIStorageCapacity: even though the etcd test verifies we're still persisting v1beta1 /kind bug /priority important-soon /sig api-machinery cc Originally posted by in
/cc
cc
hi I have found that on points to but here the points to here is screenshot from debug console ! As I am still new to the codebase, I am trying to understand why it is happening so. But we can say that because of this reason, test case below does not failed because the expected hash we put in file matches correctly with the onc we create in at line 404 I also believe that the value corresponding to key has been wrongly put same as Correct values are as follows. One caveat here is, hashes are calculated with appending resource but the key for lookup is based on resource . I hope this information will help in locating the issue. Meanwhile, I will also try to look why in is pointing to even though we override.
/assign
The problem reported by this issue should not exist, on apiserver startup, we can get the right overrided version v1beta1 for the csistoragecapacities from , since the code below overrided the storage version ! But in the test , it did not override the storage version, see the code below so this test case reports v1 CSIStorageCapacity is right. do you think that we should change the testcase to override the storage version again to keep consistent and dispel the confusion for StorageVersionHash ?
Yes you are right. If we add below testcase will surely fail as the hash will not match in this case.

So, we need to update hash in to
ah, good catch... so this is a test-only issue, but still one we need to resolve, so the unit test reflects reality of storage versions when you actually start a kube-apiserver
As this pr would remove the storageversion override so after this pr merged, the testcase would be just right, then we don't need another pr to fix the testcase again, right?
But alteast we should update the correct hash value corresponds to in
the test would still be exercising preferred version ordering without any overrides, which would still be incorrect, and would resurface problems as soon as another override is
/triage accepted


Question: } func (util *ISCSIUtil) DetachDisk(c iscsiDiskCleaner, mntPath string) error { <del> device, cnt, err := mount.GetDeviceNameFromMount(c.mounter, mntPath) </del> <ins> _, cnt, err := mount.GetDeviceNameFromMount(c.mounter, mntPath) </ins> if err != nil { glog.Errorf("iscsi detach disk: failed to get device from mnt: %snError: %v", mntPath, err) return err

During iscsi volume cleanup, kubelet reads from and retrieves a list of device paths where iscsi volume mounts on. The device path was expected to be in the form of . But currently the device path in is in the form of , thus iscsi panicked . I suspect this panic started to happen after switched to use from . seems to always resolve device path (see output below), and thus the expected iscsi configuration in the device path no longer exists.


Question: "go.uber.org/zap" "go.uber.org/zap/zapcore" <ins> "k8s.io/apimachinery/pkg/types" </ins> logsapi "k8s.io/component-base/logs/api/v1" )

should implement . This ensures that it's use as value in a key/value pair emits a struct and not the string. /ref
This depends on logr 1.2.0, which will be in /wg structure-logging /sig instrumentation
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : This depends on logr 1.2.0, which will be in /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on this :) /assign
/wg structured-logging
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /lifecycle frozen


Question: results[i] = make(schedulerapi.HostPriorityList, len(nodes)) } } processNode := func(index int) { nodeInfo := nodeNameToInfo[nodes[index].Name] var err error

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: A failed Prioritize Map Function caused Scheduler break down. What you expected to happen: Pod remains Pending and describe pod will show errors of failed functions. How to reproduce it (as minimally and precisely as possible): It happens when some prioritize's map functions return with error. Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
As shown above, if priorityConfigs[i].Map(pod, meta, nodeInfo) returns with error, there will be no chance for results[i + 1] to initialize its value, thus causing panic when calling config.Reduce(pod, meta, nodeNameToInfo, results[i+1])
/sig scheduling
I opened a PR to initialize results with real host name instead of empty string.


Question: return true, fmt.Errorf("fail to get etcd pod's hash: %v", err) } <del> // Write the updated etcd static Pod manifest into the temporary directory </del> <ins> // Write the updated etcd static Pod manifest into the temporary directory, at this point no etcd change // has occured in any aspects. </ins> if err := etcdphase.CreateLocalEtcdStaticPodManifestFile(pathMgr.TempManifestDir(), cfg); err != nil { <del> return true, rollbackEtcdData(cfg, fmt.Errorf("error creating local etcd static pod manifest file: %v", err), pathMgr) </del> <ins> return true, fmt.Errorf("error creating local etcd static pod manifest file: %v", err) </ins> } // Perform etcd upgrade using common to all control plane components function if err := upgradeComponent(constants.Etcd, waiter, pathMgr, cfg, beforeEtcdPodHash, recoverManifests); err != nil { <del> return true, rollbackEtcdData(cfg, err, pathMgr) </del> <ins> // Since etcd upgrade component failed, the old manifest has been restored // now we need to check the heatlth of etcd cluster if it came back up with old manifest if _, err := etcdCluster.GetEtcdClusterStatus(); err != nil { // At this point we know that etcd cluster is dead and it is safe to copy backup datastore and to rollback old etcd manifest if err := rollbackEtcdData(cfg, fmt.Errorf("etcd cluster is not healthy after upgrade: %v rolling back", err), pathMgr); err != nil { // Even copying back datastore failed, no options for recovery left, bailing out return true, fmt.Errorf("fatal error upgrading local etcd cluster: %v, the backup of etcd database is stored here:(%s)", err, backupEtcdDir) } // Old datastore has been copied, rolling back old manifests if err := rollbackOldManifests(recoverManifests, err, pathMgr, true); err != nil { // Rolling back to old manifests failed, no options for recovery left, bailing out return true, fmt.Errorf("fatal error upgrading local etcd cluster: %v, the backup of etcd database is stored here:(%s)", err, backupEtcdDir) } // Since rollback of the old etcd manifest was successful, checking again the status of etcd cluster if _, err := etcdCluster.GetEtcdClusterStatus(); err != nil { // Nothing else left to try to recover etcd cluster return true, fmt.Errorf("fatal error upgrading local etcd cluster: %v, the backup of etcd database is stored here:(%s)", err, backupEtcdDir) } return true, fmt.Errorf("fatal error upgrading local etcd cluster: %v, rolled the state back to pre-upgrade state", err) } // Since etcd cluster came back up with the old manifest return true, fmt.Errorf("fatal error when trying to upgrade the etcd cluster: %v, rolled the state back to pre-upgrade state", err) </ins> } // Checking health state of etcd after the upgrade <del> etcdStatus, err = etcdCluster.GetEtcdClusterStatus() if err != nil { return true, rollbackEtcdData(cfg, fmt.Errorf("etcd cluster is not healthy after upgrade: %v rolling back", err), pathMgr) </del> <ins> if _, err = etcdCluster.GetEtcdClusterStatus(); err != nil { // Despite the fact that upgradeComponent was sucessfull, there is something wrong with etcd cluster // First step is to restore back up of datastore if err := rollbackEtcdData(cfg, fmt.Errorf("etcd cluster is not healthy after upgrade: %v rolling back", err), pathMgr); err != nil { // Even copying back datastore failed, no options for recovery left, bailing out return true, fmt.Errorf("fatal error upgrading local etcd cluster: %v, the backup of etcd database is stored here:(%s)", err, backupEtcdDir) } // Old datastore has been copied, rolling back old manifests if err := rollbackOldManifests(recoverManifests, err, pathMgr, true); err != nil { // Rolling back to old manifests failed, no options for recovery left, bailing out return true, fmt.Errorf("fatal error upgrading local etcd cluster: %v, the backup of etcd database is stored here:(%s)", err, backupEtcdDir) } // Since rollback of the old etcd manifest was successful, checking again the status of etcd cluster if _, err := etcdCluster.GetEtcdClusterStatus(); err != nil { // Nothing else left to try to recover etcd cluster return true, fmt.Errorf("fatal error upgrading local etcd cluster: %v, the backup of etcd database is stored here:(%s)", err, backupEtcdDir) } return true, fmt.Errorf("fatal error upgrading local etcd cluster: %v, rolled the state back to pre-upgrade state", err) </ins> } return false, nil

kubeadm etcd upgrade modifying recovery steps /assign /sig cluster-lifecycle /kind bug


Question: return nil, err } if incluster == nil { <del> return nil, fmt.Errorf("cluster doesn't provide client-ca-file") </del> <ins> return &s.ClientCert, ignorableError{fmt.Errorf("cluster doesn't provide client-ca-file in configmap/%s in %s, so client certificate authentication to extension api-server won't work.", authenticationConfigMapName, authenticationConfigMapNamespace)} </ins> } return incluster, nil }

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: When I start the core API server with , it populates the ConfigMap in namespace with the requestheader CA. This allows the extension API server to dynamically discover the CA and use it for validating the client certificate of aggregator layer. See for documentation However, if the core API server doesn't use certificate based authentication for it's clients i.e. flag isn't set in core API server then the extension API server fails to start with error message I believe this is coming from bootstrap code in extension API server . What you expected to happen: Extension API server should be able to start and not require that both and be present in the ConfigMap. This allows Kubernetes clusters which don't use certificate based authentication for core API server to be able to dynamically discover the requestheader-ca instead of having to mount it as a secret. How to reproduce it (as minimally and precisely as possible): Create a Kubernetes cluster which uses non certificate based authentication for core API server like Webhook based authentication. Example: You can launch an Amazon EKS cluster which uses Webhook based authentication following steps outlined . Setup aggregator layer by adding flags mentioned . CA certificate and client certificate for aggregator layer can be generated using and . Deploy metrics server to cluster following steps in Deployment section of README Get logs of metrics server pod and it should be failing with error message above Anything else we need to know?: It's possible there is a reason which is not evident to me as to why extension API server would require both client-ca-file and requestheader-client-ca-file to be present in the ConfigMap. Environment: Kubernetes version (use ): 1.10.3 Cloud provider or hardware configuration: Amazon EKS OS (e.g. from /etc/os-release): Kubernetes control plane and nodes = Amazon Linux 2 (2017.12) LTS Release Candidate (4.14.42).
/sig api-machinery /cc
since it looks like the at-mention didn't take
/sig auth /assign
Thanks Jordan. This is something we are interested in contributing to and ask for being cherrypicked to patch releases of 1.10.x and 1.11.x. That conversation can wait once we have the PR out.

Quick update: I'm working on sending a PR for this issue by this week
- Now that this is merged, can we open up the possibility of cherry picking to 1.10.x and 1.11.X please?
sure, opened , ,


Question: tc.runTest(t) } <ins> func TestScaleUpIgnoresFailedPods(t *testing.T) { tc := testCase{ minReplicas:          2, maxReplicas:          6, initialReplicas:      2, desiredReplicas:      4, CPUTarget:            30, CPUCurrent:           60, verifyCPUCurrent:     true, reportedLevels:       []uint64{500, 700}, reportedCPURequests:  []resource.Quantity{resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0"), resource.MustParse("1.0")}, reportedPodReadiness: []v1.ConditionStatus{v1.ConditionTrue, v1.ConditionTrue, v1.ConditionFalse, v1.ConditionFalse}, reportedPodPhase:     []v1.PodPhase{v1.PodRunning, v1.PodRunning, v1.PodFailed, v1.PodFailed}, useMetricsAPI:        true, } tc.runTest(t) } </ins> func TestScaleUpDeployment(t *testing.T) { tc := testCase{ minReplicas:         2,

/kind bug What happened: HPA fails to launch additional pods due to pods in evicted state, even with utilization significantly over target. What you expected to happen: When HPA pods reach the target utilization, new pods should be allocated and provisioned. How to reproduce it (as minimally and precisely as possible): Launch a HPA, let the pods fill the disk on the host which will trigger evictions due to "Pod The node was low on resource: [DiskPressure]", HPA will exceed target and not allocate additional pods. Anything else we need to know?: Seems related to and though those are not specific to HPA. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): cos-stable-57-9202-64-0 kernel commit id Kernel (e.g. ):Linux gke-prod1-use1b-v2--35s4 4.4.70+ SMP Mon Sep 18 12:43:08 PDT 2017 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
/sig scheduling
Are there any workarounds while we wait for a fix? We have ran into this on a 1.8.4 GKE cluster.
/sig autoscaling /remove-sig scheduling
I am thinking of fixing this by excluding pods with Phase=Failed from the unreadyPods that are taken into account. It seems the right thing to do as they will not become running again. Do you see any issues?
that seems decent -- anything that will never be running again should not be factored into our calculations.
/status approved-for-milestone Adding to milestone -- if the HPA isn't functioning in the case of evictions, that seems like a decently severe HPA bug.
oops, missing priority was my fault, moving back into milestone: /priority critical-urgent /remove-priority important-soon
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues.

Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: }() return framework.NewStatus(framework.Wait, ""), 3 * time.Second } <del> if pp.allowPermit && pod.Name != "waiting-pod" { return nil, 0 } </del> if pp.waitAndRejectPermit || pp.waitAndAllowPermit { <del> if pod.Name == "waiting-pod" { </del> <ins> if pp.waitingPod == "" || pp.waitingPod == pod.Name { pp.waitingPod = pod.Name </ins> return framework.NewStatus(framework.Wait, ""), 30 * time.Second } <del> // This is the signalling pod, wait until the waiting-pod is actually waiting and then either reject or allow it. wait.Poll(10*time.Millisecond, 30*time.Second, func() (bool, error) { w := false pp.fh.IterateOverWaitingPods(func(wp framework.WaitingPod) { w = true }) return w, nil }) </del> if pp.waitAndRejectPermit { <ins> pp.rejectingPod = pod.Name </ins> pp.fh.IterateOverWaitingPods(func(wp framework.WaitingPod) { wp.Reject(fmt.Sprintf("reject pod %v", wp.GetPod().Name)) }) return framework.NewStatus(framework.Unschedulable, fmt.Sprintf("reject pod %v", pod.Name)), 0 } if pp.waitAndAllowPermit { <ins> pp.allowingPod = pod.Name </ins> pp.allowAllPods() return nil, 0 }

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestCoSchedulingWithPermitPlugin Testgrid link: Reason for failure: /sig scheduling
Could be related
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another run
Hmmm I think test plugin is no longer correct: After my change is executed at the end of the scheduling cycle, this means there can be only one happening at any given moment. If there is a race where "signaling-pod" is scheduled before "waiting-pod", then "signaling-pod" will wait 30s in in for the "waiting-pod" to become waiting, but this cannot happen. Basically "signaling-pod" is allowing "waiting-pod" before "waiting-pod" is being scheduled. I think test and test plugin could be changed, so there are no longer two pods of different kind. Instead first pod that arrives to the is considered "waiting-pod" and second pod is always "signaling-pod" that will either reject or allow first pod. This way we do not have to care about order in which scheduler will process these pods. I think it is also consistent with how actual co-scheduling will work.
Can you first confirm that this is what is happening? you could increase the log level to 3 to see the log lines in and . I feel like if that's the case, we have a separate bug with the event handlers or sorting. Other than that, I agree that the implementation of the test's shouldn't have waits. cc
My hope would be that the pod is temporarily unschedulable due to nodes still being synced.
Where you able to confirm the reason of the flake?
not yet, what is the easiest way to run this test multiple times? Should I try to reproduce this locally or create some temporary PR just for this purpose?

I wrote PR for the test fix to remove sleep from the PermitPlugin, it had to be done sooner than later regardless of the cause of flakiness.
Understanding the reason of the flake is important to make sure we are fixing the right thing and that we won't have future flakes. Please try to run the tests locally about 10 times.
It should be something like: If that doesn't work, run first
Flake point:
From the logs provided by you can tell that the problem is a race between "signalling-pod" and "waiting-pod" and waiting inside the Permit(), as I mentioned in this issue. BLUF: In the first case of the TestCoSchedulingWithPermitPlugin "waiting-pod" and "signalling-pod" pods are being retried many times, most of the time "signalling-pod" is rejecting already waiting "waiting-pod", but sometimes race can happen and "signalling-pod" will be processed by the scheduler before rejected "waiting-pod" (there is no guarantees whether rejecting pod or rejected pod will end up in the queue first). When this happens "signalling-pod" waits 30s inside the Permit() for "waiting-pod" to become waiting and blocks the scheduler, meanwhile second test case starts, "signalling-pod" eventually stops waiting for "waiting-pod" and gets bound, after that "waiting-pod" cannot be allowed, which leads to a failure of the second test case. My PR should fix this issue.
I feel like we should avoid sharing the scheduler & permitPlugin instances among the 2 subtests, which can then solve the flake?
I think this flake (in different variations) can happen as long as test depends on the order in which signalling and waiting pod enter the permit plugin.
I believe the PR might fix the issue. But since the reason for the flake is that one subtest is corrupting the other, it's possible that we might surface other flakes. That's why it's important that you run the test several times to check for other flakes. It several tests pass, let's proceed with this PR and leave a TODO to remove the dependency between the subtests.
I ran 10+ times on my branch with the fix and had no issues.


Question: "The order of plugins in which they are passed to this flag does not matter. "+ "Comma-delimited list of: "+strings.Join(a.GenericAdmission.Plugins.Registered(), ", ")+".") fs.MarkDeprecated("admission-control", "Use --enable-admission-plugins or --disable-admission-plugins instead. Will be removed in a future version.") <ins> fs.Lookup("admission-control").Hidden = false </ins> a.GenericAdmission.AddFlags(fs) }

After merged, we should also show deprecated kube-apiserver flags.
/sig api-machinery


Question: "net/url" "path" "path/filepath" <ins> "regexp" </ins> "runtime" "strings" "sync"

With version 1.4.7 I could use a datastore that is in a cluster as a vsphere volume as . When I upgraded to 1.6.0, this same exact path does not work and throws a error. I'm struggling to find what changed, but am guessing it's difference in how the datastore is being looked up in vcenter. Has anyone had luck with this?
We have verified creating pod using static PV and dynamic provisioning, both works fine with Can you post here? cc:
Did that include using a datastore that was in a datastore cluster? That seemed to be working before because the datastore previously (before 1.5) was read out of the global config, which supported datastore path with cluster name. But post 1.5 datastore is parsed from the entry, which if you include the path to the cluster it gets you past the datastore lookup, but then when you go to add the disk it fails because that is not valid to vCenter. It's kind of this weird mixed mode, because to look up the datastore you need the path including the cluster, but to create the file you need the path without the cluster. I presume the PV is working because they use the cloud config values? I was not using a PV but putting a volumePath directly in my deployment yaml. Should I be using a PV to do this? If the deployment is deleted and re-created how do I guarantee that my pod would get the same PV claim? That was why I did not go down that path originally. I'm pretty sure I have narrowed down where in the provider this is happening, but didn't see a good path to get it to work. I have, admittantly, not tried PV/PVCs yet, due to the concern raised above, but could try those to see. The pertinent part of the deployment yaml: The cloud config:
Just confirmed I have the same issue using a PV and PVC....
Here it needs it in format: (which it gets on line 791 by parsing . But here it needs it in true format, without the datastorecluster: Therefore if you do it gets you past line 796 but fails on 802. But if you do then you don't get past line 796.
Thanks for detailed look up. This is the commit which caused this regression. We will work on the fix.

cc:
Ah yes, I see that now. What is the lifecycle of a dynamic disk? Does that disk live on after the PVC is released? After the PV is deleted? It seems like I may just not fully be exercising the power of the volume plugins as well. I would still have the issue with existing volumes as they have data that I need to maintain, but would be interesting for new volumes to look into these more.
RECLAIMPOLICY for PV created using Dynamic Provisioning is Delete PV will get deleted if you delete PVC. Disk also gets deleted from datastore.


Question: val := int64(10) obj.PodMaxBackoffSeconds = &val } <ins> // Enable profiling by default in the scheduler if obj.EnableProfiling == nil { enableProfiling := true obj.EnableProfiling = &enableProfiling } // Enable contention profiling by default if profiling is enabled if *obj.EnableProfiling && obj.EnableContentionProfiling == nil { enableContentionProfiling := true obj.EnableContentionProfiling = &enableContentionProfiling } </ins> }

What would you like to be : Enable profiling by default in the scheduler. We can do this by defaulting the CC flags to true. Why is this needed: It is useful when debugging the performance of a running cluster. API server already enables profiling by default. /sig scheduling /help /milestone v1.17
/assign /remove-help


Question: // Trust that if we get a region from configuration or AWS metadata that it is valid, // and register ECR providers <del> RecognizeRegion(regionName) </del> <ins> recognizeRegion(regionName) </ins> if !cfg.Global.DisableStrictZoneCheck { valid := isRegionValid(regionName)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: From /kind cleanup What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cloud-provider /assign


Question: File Path passed as a flag on the command line. This file is rechecked every 20 seconds (configurable with a flag). HTTP endpoint HTTP endpoint passed as a parameter on the command line. This endpoint is checked every 20 seconds (also configurable with a flag). <del> etcd server The Kubelet will reach out and do a watch on an etcd server. The etcd path that is watched is /registry/hosts/$(hostname -f). As this is a watch, changes are noticed and acted upon very quickly. </del> <ins> etcd server The Kubelet will reach out and do a watch on an etcd server. The etcd path that is watched is /registry/hosts/$(uname -n). As this is a watch, changes are noticed and acted upon very quickly. </ins> HTTP server The kubelet can also listen for HTTP and respond to a simple API (underspec'd currently) to submit a new manifest.

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: // NewNamedReflector same as NewReflector, but with a specified name for logging func NewNamedReflector(name string, lw ListerWatcher, expectedType interface{}, store Store, resyncPeriod time.Duration) *Reflector { <ins> realClock := &clock.RealClock{} </ins> r := &Reflector{ name:          name, listerWatcher: lw, store:         store, <del> period:        time.Second, resyncPeriod:  resyncPeriod, clock:         &clock.RealClock{}, </del> <ins> // We used to make the call every 1sec (1 QPS), the goal here is to achieve ~98% traffic reduction when // API server is not healthy. With these parameters, backoff will stop at [30,60) sec interval which is // 0.22 QPS. If we don't backoff for 2min, assume API server is healthy and we reset the backoff. backoffManager: wait.NewExponentialBackoffManager(800*time.Millisecond, 30*time.Second, 2*time.Minute, 2.0, 1.0, realClock), resyncPeriod:   resyncPeriod, clock:          realClock, </ins> } r.setExpectedType(expectedType) return r

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We recently had a production issue where one of our large cluster (~3k nodes) has it's API server OOMKilled. This caused all kubelets' connection to master broken, and caused a global synchronization of kubelet's reconnecting to master. Due to the burst of API call (18K QPS, 7x than normal), API server went into even worse condition, and caused more replicas to get OOMKilled. Current reflector impl has fixed 1sec backoff before calling next , which shall be the root cause of such instability. What you expected to happen: reflectors to backoff with jitter How to reproduce it (as minimally and precisely as possible): a cluster with a few thousands of nodes api server and reboot it Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign /sig api-machinery scalability
/cc


Question: `pod1 (and 3 other pods): message`, }, }, <del> // TODO: test for prioritizing evaluating pods from unique controllers </del> <ins> { name:                 "prioritized pods", exemptRuntimeClasses: []string{"runtimeclass1"}, newLabels:            map[string]string{api.EnforceLevelLabel: string(api.LevelRestricted)}, oldLabels:            map[string]string{api.EnforceLevelLabel: string(api.LevelBaseline)}, pods: []*corev1.Pod{ // ensure exempt pods don't use up the limit of evaluated pods {ObjectMeta: metav1.ObjectMeta{Name: "exemptpod1", Annotations: map[string]string{"error": "message1"}}, Spec: corev1.PodSpec{RuntimeClassName: pointer.String("runtimeclass1")}}, {ObjectMeta: metav1.ObjectMeta{Name: "exemptpod2", Annotations: map[string]string{"error": "message1"}}, Spec: corev1.PodSpec{RuntimeClassName: pointer.String("runtimeclass1")}}, {ObjectMeta: metav1.ObjectMeta{Name: "exemptpod3", Annotations: map[string]string{"error": "message1"}}, Spec: corev1.PodSpec{RuntimeClassName: pointer.String("runtimeclass1")}}, {ObjectMeta: metav1.ObjectMeta{Name: "exemptpod4", Annotations: map[string]string{"error": "message1"}}, Spec: corev1.PodSpec{RuntimeClassName: pointer.String("runtimeclass1")}}, // ensure replicas from the same controller don't use up limit of evaluated pods {ObjectMeta: metav1.ObjectMeta{Name: "replicaset1pod1", Annotations: map[string]string{"error": "replicaset1error"}, OwnerReferences: []metav1.OwnerReference{{UID: types.UID("1"), Controller: pointer.Bool(true)}}}}, {ObjectMeta: metav1.ObjectMeta{Name: "replicaset1pod2", Annotations: map[string]string{"error": "replicaset1error"}, OwnerReferences: []metav1.OwnerReference{{UID: types.UID("1"), Controller: pointer.Bool(true)}}}}, {ObjectMeta: metav1.ObjectMeta{Name: "replicaset1pod3", Annotations: map[string]string{"error": "replicaset1error"}, OwnerReferences: []metav1.OwnerReference{{UID: types.UID("1"), Controller: pointer.Bool(true)}}}}, {ObjectMeta: metav1.ObjectMeta{Name: "replicaset1pod4", Annotations: map[string]string{"error": "replicaset1error"}, OwnerReferences: []metav1.OwnerReference{{UID: types.UID("1"), Controller: pointer.Bool(true)}}}}, {ObjectMeta: metav1.ObjectMeta{Name: "replicaset2pod1", Annotations: map[string]string{"error": "replicaset2error"}, OwnerReferences: []metav1.OwnerReference{{UID: types.UID("2"), Controller: pointer.Bool(true)}}}}, {ObjectMeta: metav1.ObjectMeta{Name: "replicaset2pod2", Annotations: map[string]string{"error": "replicaset2error"}, OwnerReferences: []metav1.OwnerReference{{UID: types.UID("2"), Controller: pointer.Bool(true)}}}}, {ObjectMeta: metav1.ObjectMeta{Name: "replicaset2pod3", Annotations: map[string]string{"error": "replicaset2error"}, OwnerReferences: []metav1.OwnerReference{{UID: types.UID("2"), Controller: pointer.Bool(true)}}}}, {ObjectMeta: metav1.ObjectMeta{Name: "replicaset2pod4", Annotations: map[string]string{"error": "replicaset2error"}, OwnerReferences: []metav1.OwnerReference{{UID: types.UID("2"), Controller: pointer.Bool(true)}}}}, // ensure unique pods are prioritized before additional replicas {ObjectMeta: metav1.ObjectMeta{Name: "uniquepod1", Annotations: map[string]string{"error": "uniquemessage1"}}}, {ObjectMeta: metav1.ObjectMeta{Name: "uniquepod2", Annotations: map[string]string{"error": "uniquemessage2"}}}, {ObjectMeta: metav1.ObjectMeta{Name: "uniquepod3", Annotations: map[string]string{"error": "uniquemessage3"}}}, {ObjectMeta: metav1.ObjectMeta{Name: "uniquepod4", Annotations: map[string]string{"error": "uniquemessage4"}}}, }, expectAllowed:  true, expectListPods: true, expectEvaluate: api.LevelVersion{Level: api.LevelRestricted, Version: api.LatestVersion()}, expectWarnings: []string{ `new PodSecurity enforce level only checked against the first 4 of 12 existing pods`, `existing pods in namespace "test" violate the new PodSecurity enforce level "restricted:latest"`, `replicaset1pod1: replicaset1error`, `replicaset2pod1: replicaset2error`, `uniquepod1: uniquemessage1`, `uniquepod2: uniquemessage2`, }, }, </ins> } for _, tc := range testcases {

When updating policy labels on a namespace, existing pods are evaluated to see if they are compatible with the new policy. If there are too many pods, or if evaluation takes too long, we short circuit and warn that not all pods were checked. For that reason, we want to prioritize checking a variety of pods before checking essentially identical pods, so that unique warnings can be surfaced before reaching the pod count cutoff or evaluation timeout. Pods created by the same controller are most likely to be identical (e.g. replicas from a replicaset or job, copies of pods from a statefulset or daemonset, etc) cc
This is an interesting fix I'd like to take this up Can I take this up? I am fairly new to Kubernetes source code so might need some help
I am guessing this is where the logic you mentioned should be implemented right?
thanks for the interest; this is targeted at the beta version... right now, effort is going toward the alpha-blocking work for 1.22
Should I wait for the code freeze before working on this? Sorry for lame questions
If you want to start looking at it, that's fine, but it's possible the code in the ValidateNamespace function will change before we're ready to start merging 1.23 work, and there isn't really reviewer bandwidth to take a look at work-in-progress stuff for next release yet.
Sure, I'll just for code freeze before I start with implementation By the time I'll just assign this to myself /assign
Does it make sense to take it up now? I think we already have a couple of RC builds after the code freeze
sure, it's fine to start work in a branch in your fork and have a WIP PR open, even before master reopens for 1.23 work
Awesome, I'll start working on it Thank you
Hello! Are you still working on this? If you don't have time, I can take this issue. If not, maybe I can give some help
Since this was not an urgent issue I was working on some other issues last week I have started on this, Will be raising a PR maybe today or tomorrow I Will ask you if I get stuck Thank you


Question: kube::util::wait_for_url() { local url=$1 local prefix=${2:-} <del> local wait=${3:-0.2} local times=${4:-10} </del> <ins> local wait=${3:-0.5} local times=${4:-25} </ins> which curl >/dev/null || { kube::log::usage "curl must be installed"

Example:
Is kubelet healthz coupled with the docker daemon being accessible? ?
That particular error is coming from cAdvisor and should not be causing the test to block. The Kubelet does wait up to 5min for Docker to come up though. Do we use a real Docker in these tests?
Shippable is running on different os dist? ContainerOsVersion:Ubuntu precise?
Yes today kubelet healthz coupled with docker daemon. We even periodically healthcheck docker and kubelet, and restart them even there are issues.
Another example:
bumping priority because we have to get these flaky tests fixed.
This has been marked P0 for 2 months but hasn't had any updates. Is this still happening on Shippable? If so, can we just increase a timeout in the test?
This is ridiculous, I'll fix it myself.


Question: "fmt" "io/ioutil" "os" <ins> "path" </ins> "path/filepath" "reflect" "strings"

Apply this patch: Then run: make test WHAT=./pkg/volume/git_repo The test fails: Test should pass. See above. The full loop is: The first problem is that fakeOutputs is capturing , so all the funcs on that list use the from the final iteration, not the one from the iteration with the [1] == "clone". But then this problem appears to be covering up a different problem in the test scenarios themselves. I looked at it for a little bit but I haven't puzzled out what the test is trying to do exactly. Hopefully someone with more knowledge of the test can take a look. It seems like a buggy test, accidentally passing due to this other loopvar sharing bug. HEAD N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority important-soon /cc Thanks for the report and the detailed explanation
/assign
I wonder if it could be beneficial to invest some time and effort to identify a maximum of loops that contain the loopvar sharing bug maybe with comment and a todo ?


Question: return fmt.Errorf("cannot set --all and --field-selector at the same time") } <ins> if o.GracePeriod == 0 && !o.ForceDeletion && !o.WaitForDeletion { // With the explicit --wait flag we need extra validation for backward compatibility return fmt.Errorf("--grace-period=0 must have either --force specified, or --wait to be set to true") } </ins> switch { case o.GracePeriod == 0 && o.ForceDeletion: fmt.Fprintf(o.ErrOut, "warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.n")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc


Question: } if ep.ip == rs.providerAddress { providerAddress = rs.providerAddress <ins> containsNodeIP = true </ins> } } if len(providerAddress) == 0 { <del> klog.Errorf("Could not find provider address for %s", ep.ip) </del> <ins> klog.Infof("Could not find provider address for %s. Assuming it is a public IP", ep.ip) </ins> providerAddress = proxier.nodeIP.String() containsPublicIP = true } hnsEndpoint := &endpointsInfo{ ip:              ep.ip, isLocal:         false,

What happened: Executing inside a Windows Pod, but get the following msg:  What you expected to happen: Executing inside a Windows Pod, could view: How to reproduce it (as minimally and precisely as possible): Flannel mode to create a Cluster with one control plane (via RKE) and one Windows Node (via ) into the Windows Pod is fine is fine , but get timeout error Anything else we need to know?: Windows workload yaml:Nodes info:  Content of from :More researching:Deploy an Nginx workload with host port exposed, and then create an Endpoint and Service for it, finally access the vip from Windows inside:   Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): Linux: Windows: Kernel (e.g. ): Linux: Windows: Install tools: Control plane created via RKE, Windows node created by Kuberentes guide Network plugin and version (if this is a network-related bug): Flannel VxLan
/sig windows
More information If we edit the default kubernetes endpoint to point to the node’s public ip, then works
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
can you provide the Kubernetes version for the windows node? I see platform as linux in your post
Hi , it happens in 1.14, 1.15 and 1.16. You sent us a commit to fix it before. It works well. Hope we can get it fixed in upstream as well.
In which version was it fixed?

The k8s version I use is: windows Server core version: The problem still appears
The fix for the issue above is included in 1.18.1 to help further
But it doesn't work in 1.18.1
Verified in K8s 1.18. It only works in some particular Windows versions. I guess it is a Windows (OS) issue. works. doesn't work. If you are using AWS, please use instead of the current default one . CC
I do n’t think it should be a system problem, I am using Microsoft Windows Server version 1909 [OS build .778]
: we have identified a potential issue in this area last week. In order to make sure we are talking about the same issue, would it be possible to collect some logs using the instructions below, please? an elevated powershell Window on the host and run: wget -o collectlogs.ps1 the same command line, run .collectlogs.ps1 share the resulting logs folder
Here is a link to the issue I mentioned in my previous comment: Though the issue description is different, the end result is the same: is not accessible from within the POD
any news about that? I'm also running into this issue with newer Windows Versions and can't stay on build .1039
: Two fixes were merged to the master branch a couple of weeks ago. kubernetes/kubernetes kubernetes/kubernetes AFAICT there is no kubernetes releases yet that include these fixes, but I am expecting the next v1.19.x to include these.
and FYI.


Question: } return false } <ins> // WatchEventSequenceVerifier ... // manages a watch for a given resource, ensures that events take place in a given order, retries the test on failure //   testContext         cancelation signal across API boundries, e.g: context.TODO() //   dc                  sets up a client to the API //   resourceType        specify the type of resource //   namespace           select a namespace //   resourceName        the name of the given resource //   listOptions         options used to find the resource, recommended to use listOptions.labelSelector //   expectedWatchEvents array of events which are expected to occur //   scenario            the function to run func WatchEventSequenceVerifier(testContext context.Context, dc dynamic.Interface, resourceType schema.GroupVersionResource, namespace string, resourceName string, listOptions metav1.ListOptions, expectedWatchEvents []watch.Event, scenario func(*watchtools.RetryWatcher) []watch.Event) { listWatcher := &cache.ListWatch{ WatchFunc: func(listOptions metav1.ListOptions) (watch.Interface, error) { return dc.Resource(resourceType).Namespace(namespace).Watch(testContext, listOptions) }, } resourceWatch, err := watchtools.NewRetryWatcher("1", listWatcher) ExpectNoError(err, "Failed to create a resource watch of %v in namespace %v", resourceType.Resource, namespace) // NOTE value of 3 retries seems to make sense retries := 3 retriesLoop: for try := 1; try <= retries; try++ { // NOTE the test may need access to the events to see what's going on, such as a change in status actualWatchEvents := scenario(resourceWatch) errs := sets.NewString() // watchEventsLoop: totalValidWatchEvents := 0 actualWatchEventsHasDelete := false for watchEventIndex := range expectedWatchEvents { foundExpectedWatchEvent := false ExpectEqual(len(expectedWatchEvents) <= len(actualWatchEvents), true, "Error: actual watch events amount must be greater than or equal to expected watch events amount") for actualWatchEventIndex := range actualWatchEvents { if actualWatchEvents[watchEventIndex].Type == expectedWatchEvents[actualWatchEventIndex].Type { foundExpectedWatchEvent = true } if actualWatchEvents[actualWatchEventIndex].Type == watch.Deleted && actualWatchEventsHasDelete != true { actualWatchEventsHasDelete = true } } if foundExpectedWatchEvent == false { errs.Insert(fmt.Sprintf("Watch event %v not found", expectedWatchEvents[watchEventIndex].Type)) } totalValidWatchEvents++ } if actualWatchEventsHasDelete == false { _ = dc.Resource(resourceType).Namespace(namespace).DeleteCollection(testContext, metav1.DeleteOptions{}, listOptions) } // TODO restructure failures handling if errs.Len() > 0 && try < retries { fmt.Println(fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) continue retriesLoop } ExpectEqual(errs.Len() > 0, false, fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) ExpectEqual(totalValidWatchEvents, len(expectedWatchEvents), "Error: there must be an equal amount of total valid watch events (%v) and expected watch events (%v)", totalValidWatchEvents, len(expectedWatchEvents)) } } </ins>

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

[x] Original approval issue: Start 9 June 2020 3 Flakes No Flakes for 2 weeks According to this APIsnoop query, there are still some remaining ReplicationController endpoints which are untested. a ReplicationController with a static label the ReplicationController with a new Label and updated data the ReplicationController's status /Scale the ReplicationController's Replica count to 2 Replicas the ReplicationController the ReplicationController's Status all ReplicationControllers in all Namespaces find the ReplicationControllers(1) ensure that the ReplicationController is found and is patched Namespaced ReplicationControllers(1) via a Collection with a LabelSelector Discover useragents: See all endpoints hit from this test Display test coverage change If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
Can you please add in PR
/reopen Merged. Promotion to be created
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: } return false } <ins> // WatchEventSequenceVerifier ... // manages a watch for a given resource, ensures that events take place in a given order, retries the test on failure //   testContext         cancelation signal across API boundries, e.g: context.TODO() //   dc                  sets up a client to the API //   resourceType        specify the type of resource //   namespace           select a namespace //   resourceName        the name of the given resource //   listOptions         options used to find the resource, recommended to use listOptions.labelSelector //   expectedWatchEvents array of events which are expected to occur //   scenario            the function to run func WatchEventSequenceVerifier(testContext context.Context, dc dynamic.Interface, resourceType schema.GroupVersionResource, namespace string, resourceName string, listOptions metav1.ListOptions, expectedWatchEvents []watch.Event, scenario func(*watchtools.RetryWatcher) []watch.Event) { listWatcher := &cache.ListWatch{ WatchFunc: func(listOptions metav1.ListOptions) (watch.Interface, error) { return dc.Resource(resourceType).Namespace(namespace).Watch(testContext, listOptions) }, } resourceWatch, err := watchtools.NewRetryWatcher("1", listWatcher) ExpectNoError(err, "Failed to create a resource watch of %v in namespace %v", resourceType.Resource, namespace) // NOTE value of 3 retries seems to make sense retries := 3 retriesLoop: for try := 1; try <= retries; try++ { // NOTE the test may need access to the events to see what's going on, such as a change in status actualWatchEvents := scenario(resourceWatch) errs := sets.NewString() // watchEventsLoop: totalValidWatchEvents := 0 actualWatchEventsHasDelete := false for watchEventIndex := range expectedWatchEvents { foundExpectedWatchEvent := false ExpectEqual(len(expectedWatchEvents) <= len(actualWatchEvents), true, "Error: actual watch events amount must be greater than or equal to expected watch events amount") for actualWatchEventIndex := range actualWatchEvents { if actualWatchEvents[watchEventIndex].Type == expectedWatchEvents[actualWatchEventIndex].Type { foundExpectedWatchEvent = true } if actualWatchEvents[actualWatchEventIndex].Type == watch.Deleted && actualWatchEventsHasDelete != true { actualWatchEventsHasDelete = true } } if foundExpectedWatchEvent == false { errs.Insert(fmt.Sprintf("Watch event %v not found", expectedWatchEvents[watchEventIndex].Type)) } totalValidWatchEvents++ } if actualWatchEventsHasDelete == false { _ = dc.Resource(resourceType).Namespace(namespace).DeleteCollection(testContext, metav1.DeleteOptions{}, listOptions) } // TODO restructure failures handling if errs.Len() > 0 && try < retries { fmt.Println(fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) continue retriesLoop } ExpectEqual(errs.Len() > 0, false, fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) ExpectEqual(totalValidWatchEvents, len(expectedWatchEvents), "Error: there must be an equal amount of total valid watch events (%v) and expected watch events (%v)", totalValidWatchEvents, len(expectedWatchEvents)) } } </ins>

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

[x] Original approval issue: [x] Two weeks soak end date: 8 May 2020 [x] Two weeks soak end date: 23 June 2020 This issue is created to allow edit by /wip /hold This issue proposes the writing of a new test for the API endpoints , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining ConfigMap endpoints which are untested. a ConfigMap with a static label the ConfigMap with a new Label and updated data the ConfigMap to ensure it's patched all ConfigMaps in all Namespaces find the ConfigMap(1) ensure that the ConfigMap is found and is patched Namespaced ConfigMap(1) via a Collection with a LabelSelector This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents: Display endpoints hit by the test: Show the change in coverage after the mock test: If a test with these calls gets merged, Conformance coverage will go up by 2 points
/sig testing /sig architecture /area conformance
/assign Issue created to replace
GitHub didn't allow me to assign the following users: raymonddeng99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen On hold & soaking
Reopened this issue. <detailsIn response to : /reopen On hold & soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } return false } <ins> // WatchEventSequenceVerifier ... // manages a watch for a given resource, ensures that events take place in a given order, retries the test on failure //   testContext         cancelation signal across API boundries, e.g: context.TODO() //   dc                  sets up a client to the API //   resourceType        specify the type of resource //   namespace           select a namespace //   resourceName        the name of the given resource //   listOptions         options used to find the resource, recommended to use listOptions.labelSelector //   expectedWatchEvents array of events which are expected to occur //   scenario            the function to run func WatchEventSequenceVerifier(testContext context.Context, dc dynamic.Interface, resourceType schema.GroupVersionResource, namespace string, resourceName string, listOptions metav1.ListOptions, expectedWatchEvents []watch.Event, scenario func(*watchtools.RetryWatcher) []watch.Event) { listWatcher := &cache.ListWatch{ WatchFunc: func(listOptions metav1.ListOptions) (watch.Interface, error) { return dc.Resource(resourceType).Namespace(namespace).Watch(testContext, listOptions) }, } resourceWatch, err := watchtools.NewRetryWatcher("1", listWatcher) ExpectNoError(err, "Failed to create a resource watch of %v in namespace %v", resourceType.Resource, namespace) // NOTE value of 3 retries seems to make sense retries := 3 retriesLoop: for try := 1; try <= retries; try++ { // NOTE the test may need access to the events to see what's going on, such as a change in status actualWatchEvents := scenario(resourceWatch) errs := sets.NewString() // watchEventsLoop: totalValidWatchEvents := 0 actualWatchEventsHasDelete := false for watchEventIndex := range expectedWatchEvents { foundExpectedWatchEvent := false ExpectEqual(len(expectedWatchEvents) <= len(actualWatchEvents), true, "Error: actual watch events amount must be greater than or equal to expected watch events amount") for actualWatchEventIndex := range actualWatchEvents { if actualWatchEvents[watchEventIndex].Type == expectedWatchEvents[actualWatchEventIndex].Type { foundExpectedWatchEvent = true } if actualWatchEvents[actualWatchEventIndex].Type == watch.Deleted && actualWatchEventsHasDelete != true { actualWatchEventsHasDelete = true } } if foundExpectedWatchEvent == false { errs.Insert(fmt.Sprintf("Watch event %v not found", expectedWatchEvents[watchEventIndex].Type)) } totalValidWatchEvents++ } if actualWatchEventsHasDelete == false { _ = dc.Resource(resourceType).Namespace(namespace).DeleteCollection(testContext, metav1.DeleteOptions{}, listOptions) } // TODO restructure failures handling if errs.Len() > 0 && try < retries { fmt.Println(fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) continue retriesLoop } ExpectEqual(errs.Len() > 0, false, fmt.Errorf("invariants violated:n* %s", strings.Join(errs.List(), "n* "))) ExpectEqual(totalValidWatchEvents, len(expectedWatchEvents), "Error: there must be an equal amount of total valid watch events (%v) and expected watch events (%v)", totalValidWatchEvents, len(expectedWatchEvents)) } } </ins>

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: expected_attaches: map[string][]string{}, expected_detaches: map[string][]string{"mynode-1": {"vol1"}}, }, <del> { testName:       "CSI Migration", volName:        "vol1", podNodeName:    "mynode-1", pvName:         "pv1", vaName:         "va1", vaNodeName:     "mynode-1", vaAttachStatus: false, csiMigration:   true, </del> <ins> { // pod is scheduled, volume is migrated, attach status:false, verify volume is marked as attached testName:               "Scheduled Pod with migrated PV", volName:                "vol1", podNodeName:            "mynode-1", pvName:                 "pv1", vaName:                 "va1", vaNodeName:             "mynode-1", vaAttachStatus:         false, csiMigration:           true, expectedASWAttachState: cache.AttachStateAttached, }, { // pod is deleted, volume is migrated, attach status:false, verify volume is marked as uncertain testName:               "Deleted Pod with migrated PV", volName:                "vol1", pvName:                 "pv1", vaName:                 "va1", vaNodeName:             "mynode-1", vaAttachStatus:         false, csiMigration:           true, expectedASWAttachState: cache.AttachStateUncertain, </ins> }, } { t.Run(tc.testName, func(t *testing.T) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Context: When a cluster has dangling s caused by restarts of kube-controller-manager (like described in ), the attach/detach controller tries to clean them up on startup (ref ). For this, it iterates over all s and finds the responsible volume plugin for retrieving the unique volume name. Problem: However, if the volume was provisioned by an in-tree plugin, then later migrated to CSI and now the / feature gate is set, the volume is skipped during , so the will stay in the system and the volume will not be detached from the undesired node and potentially block a new attach operation to a different node. Analysis: The / feature gate causes the respective in-tree volume plugins not to be registered: The spec only contains the in-tree fields (e.g or ), but not the fields: This causes the call in to find no responsible volume plugin for the , which will skip the cleanup: That's why we see a lot of logs like this on startup of kube-controller-manager and dangling s on our clusters: The attach/detach controller should cleanup dangling s for migrated in-tree plugins with unregistered in-tree plugins as well. ref , addresses a similar issue. Though, the fix doesn't come into effect, if the respective / feature gate is enabled. Kubernetes version (use ): / Cloud provider or hardware configuration: AWS, GCP
/sig storage cc
Hey, thanks for reporting this bug!! I will try to come up with a fix. BTW, wondering what's the symptom you observed except for the log. Is there any volume operation(attaching/detaching/mounting/unmounting/creation/deletion) not working? This, to my understanding, will only make dangling attachment not working properly but I guess normal operation should work, is it?
The consequence of the leftover s in our case is that volumes are not detached on infrastructure provider side when moving pods with volumes between nodes, e.g.

because of rolling updates/eviction/etc.. This in turn causes the new s on the new nodes to fail with logs similar to this one: (we observe the same for GCP) I can also work on a fix tomorrow, if you want. I think, we should be able to reuse the fix from and additionally determine the from or similar, instead of only relying on .
That would be nice! Really appreciate it. Yep, sounds like a good idea. If you could also help to verify the fix in your environment that would be awesome
Ok, great. Will try to come up with a fix. /assign Definitely! I'll try my best and let you know if I need some help :)
Yep, we observed this after experiencing the bug. We ended up manually creating a cleanup script to check EC2 <-K8s VA if it was actually attached or not, otherwise delete VA. My understanding is with fixed we will no longer have this issue, but it would be nice to have the garbage collection logic work for migrated/non-migrated volumes regardless
/triage accepted


Question: permitPlugin.timeoutPermit = false permitPlugin.waitAndRejectPermit = false permitPlugin.waitAndAllowPermit = true <del> permitPlugin.allowPermit = true </del> lowPriority, highPriority := int32(100), int32(300) resourceRequest := v1.ResourceRequirements{Requests: v1.ResourceList{

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestCoSchedulingWithPermitPlugin Testgrid link: Reason for failure: /sig scheduling
Could be related
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another run
Hmmm I think test plugin is no longer correct: After my change is executed at the end of the scheduling cycle, this means there can be only one happening at any given moment. If there is a race where "signaling-pod" is scheduled before "waiting-pod", then "signaling-pod" will wait 30s in in for the "waiting-pod" to become waiting, but this cannot happen. Basically "signaling-pod" is allowing "waiting-pod" before "waiting-pod" is being scheduled. I think test and test plugin could be changed, so there are no longer two pods of different kind. Instead first pod that arrives to the is considered "waiting-pod" and second pod is always "signaling-pod" that will either reject or allow first pod. This way we do not have to care about order in which scheduler will process these pods. I think it is also consistent with how actual co-scheduling will work.
Can you first confirm that this is what is happening? you could increase the log level to 3 to see the log lines in and . I feel like if that's the case, we have a separate bug with the event handlers or sorting. Other than that, I agree that the implementation of the test's shouldn't have waits. cc
My hope would be that the pod is temporarily unschedulable due to nodes still being synced.
Where you able to confirm the reason of the flake?
not yet, what is the easiest way to run this test multiple times? Should I try to reproduce this locally or create some temporary PR just for this purpose?

I wrote PR for the test fix to remove sleep from the PermitPlugin, it had to be done sooner than later regardless of the cause of flakiness.
Understanding the reason of the flake is important to make sure we are fixing the right thing and that we won't have future flakes. Please try to run the tests locally about 10 times.
It should be something like: If that doesn't work, run first
Flake point:
From the logs provided by you can tell that the problem is a race between "signalling-pod" and "waiting-pod" and waiting inside the Permit(), as I mentioned in this issue. BLUF: In the first case of the TestCoSchedulingWithPermitPlugin "waiting-pod" and "signalling-pod" pods are being retried many times, most of the time "signalling-pod" is rejecting already waiting "waiting-pod", but sometimes race can happen and "signalling-pod" will be processed by the scheduler before rejected "waiting-pod" (there is no guarantees whether rejecting pod or rejected pod will end up in the queue first). When this happens "signalling-pod" waits 30s inside the Permit() for "waiting-pod" to become waiting and blocks the scheduler, meanwhile second test case starts, "signalling-pod" eventually stops waiting for "waiting-pod" and gets bound, after that "waiting-pod" cannot be allowed, which leads to a failure of the second test case. My PR should fix this issue.
I feel like we should avoid sharing the scheduler & permitPlugin instances among the 2 subtests, which can then solve the flake?
I think this flake (in different variations) can happen as long as test depends on the order in which signalling and waiting pod enter the permit plugin.
I believe the PR might fix the issue. But since the reason for the flake is that one subtest is corrupting the other, it's possible that we might surface other flakes. That's why it's important that you run the test several times to check for other flakes. It several tests pass, let's proceed with this PR and leave a TODO to remove the dependency between the subtests.
I ran 10+ times on my branch with the fix and had no issues.


Question: for _, err := range validateDeprecationWarning(version.Deprecated, version.DeprecationWarning) { allErrs = append(allErrs, field.Invalid(fldPath.Child("deprecationWarning"), version.DeprecationWarning, err)) } <ins> opts = suppressExpressionCostForUnchangedSchema(opts, version.Name) </ins> allErrs = append(allErrs, validateCustomResourceDefinitionValidation(ctx, version.Schema, statusEnabled, opts, fldPath.Child("schema"))...) allErrs = append(allErrs, ValidateCustomResourceDefinitionSubresources(version.Subresources, fldPath.Child("subresources"))...) for i := range version.AdditionalPrinterColumns {

We were using cel-go v0.12.7 in Kubernetes 1.27 release and cel-go v0.16.0 in Kubernetes 1.28 release. There are many changes from cel-go side between those two releases and some may break backward compatible of CRD validation rules. An example would be the estimated cost for CRD validation rules(Thanks to who brought this to our attention with the following example). In the example CRD validation rules below: It will pass perfect in Kubernetes 1.27 version but fail in Kubernetes 1.28 with error: This is due to the estimator fix happened in cel-go(, ) The changes should not cause backward incompatible of CRD validation rules. OR a stability test suite should be in place to prevent the changes causing backward compatible issue before GA. Apply the above CRD in Kubernetes 1.27 and 1.28, 1.27 cluster should pass and 1.28 cluster should fail. Since the fix in cel-go side makes sense to me and the feature is in Beta now, I am not quite sure how strict we should be on backward compatible. cc As a followup, a stability test suite for estimate cost is worth adding before GA to make sure any future changes be caught before the feature promoted to GA. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi I agree that the changes in cel-go should not break the backward compatibility of CRD validation rules, especially for users who have already defined and deployed their custom resources using the previous version of Kubernetes. One suggestion I have is to use a version flag for the CEL library, so that users can specify which version of cel-go they want to use for their CRD validation rules. This way, they can avoid unexpected errors or behavior changes when upgrading their Kubernetes cluster.

For example, they can add something like in their CRD schema to indicate that they want to use the old version of cel-go.
Another suggestion I have is to use a cost limit parameter for the CEL library, so that users can adjust the budget for their CRD validation rules. This way, they can fine-tune the trade-off between performance and complexity of their rules. For example, they can add something like in their CRD schema to indicate that they want to allow more expensive rules.
I agree this is the type of bug fix that is legitimate to make during beta. when the size changes came in, did our existing size estimation test coverage notice this cost increased? If not, is there a systemic change we could make to exercise each type of estimation combination so we have visibility to any future changes. Existing persisted CRD objects should continue to work as-is on upgrade, but the CEL rules and/or schema would be required to fit within fixed cost estimates before making future updates/changes to the CEL rules there should be unit tests ensuring updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates, but can you verify that is the case? That's not workable, both from an implementation perspective and from a fragmentation perspective. We explicitly don't want to allow opting into loopholes / bugs in cost estimation that allow CRD authors to make API servers do excessive amounts of work. Similar to above, the responsibility for API server resource consumption lies with the kube-apiserver admin, not the CRD author, so exposing this in the API is not something I would expect.
/assign
Thank you! I guess the next step would be: the existing test suite regarding with the cel cost evaluation. Add estimated cost tests for every single operation, function, type, etc if it's not existing now. updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates. Currently it is not working like this but would definitely help here. cc , please correct me if I'm wrong :) Those should be done before the feature goes to GA. I am not sure about the policy of a GA feature relying on alpha/beta feature is acceptable or not. is going to Beta in this release. The worst case is we have to wait further until the feature is ready.

I don't think that helps... CRD ratcheting is about validation of the custom resources. This is about validation of the CRD itself.
thanks for clarifying! Are you suggesting while updating CRD, if the rules matching what's in the existing object, we should skip cost estimation check? That makes sense. I could start a PR if my understanding aligns :)
something along those lines, though now that I think about it, other changes in the schema could affect the number of times a CEL expression could be run (dropping or increasing maxItems, maxProperties, etc) we already collect preexisting expressions when updating a CRD in order to allow already-persisted expressions (possibly persisted via a newer API server) making use of new CEL features: I think that is what I was thinking of. I would probably focus on: coverage of the edges of the cost estimation so we don't pick up changes there without knowing coarse-grained skipping of schema/cel validation for a particular version in a CRD for updates which don't change anything under that version's schema (would let someone make unrelated updates to the CRD and leave an existing version's schema / CEL rules alone)
Thank you! We have those work planned for the current cycle. /assign /assign


Question: }) ginkgo.AfterEach(func() { <ins> defer func() { if expectedNodeCondition == v1.NodeDiskPressure && framework.TestContext.PrepullImages { // The disk eviction test may cause the prepulled images to be evicted, // prepull those images again to ensure this test not affect following tests. PrePullAllImages() } }() </ins> ginkgo.By("deleting pods") for _, spec := range testSpecs { ginkgo.By(fmt.Sprintf("deleting pod: %s", spec.pod.Name)) f.PodClient().DeleteSync(spec.pod.Name, &metav1.DeleteOptions{}, 10*time.Minute) } reduceAllocatableMemoryUsage() <del> if expectedNodeCondition == v1.NodeDiskPressure && framework.TestContext.PrepullImages { // The disk eviction test may cause the prepulled images to be evicted, // prepull those images again to ensure this test not affect following tests. PrePullAllImages() } </del> ginkgo.By("making sure we can start a new pod after the test") podName := "test-admit-pod" f.PodClient().CreateSync(&v1.Pod{

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Several tests Eg. this run: Since when has it been failing: Unknown Testgrid link: Reason for failure: (happens to all the images). Pods in the e2e test suites use pull policy , , and it looks like the test during DiskPressure, resulting in kubelet removing some of the pre pulled images. Anything else we need to know: The runs after each test. This should also be the case in the test too.
/assign /sig node
These tests are now constantly failing due to missing images. This makes it hard to see if some of the stuff we do now, or have done in the last few weeks, breaks some of the tests. PR should hopefully fix this, but it is not in the 1.16 milestone, but has and . Should we get this merged into master so that we can see if if fixes the problem, or should we wait until code thaw? Any thoughts? /cc /cc /cc
/priority important-longterm
Thanks for all the work you've put into this ! I think we should add to the milestone
+1 to - thanks for your hard work :)
still has a lot of failing tests. What are the next steps? /reopen
Reopened this issue. <detailsIn response to : still has a lot of failing tests. What are the next steps? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it has over the few last weeks become quite a mess... Looks like fixed some of the issues about image eviction problems, but there are still a few left. I think I have a solution, so ill try to submit a ~PR in a few hours~ edit: will do it tomorrow. A lot of new tests started failing 8-28, and that may be related to , but I am not totally sure..
Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! ! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above).

Ill create a new issue about all the failing tests where we can track them each together. Closing this issue, as this was only filed for the flaking becuase of the evicted images. :smile: /close cc
Closing this issue. <detailsIn response to : Looks like fixed the flaking in ! The last 7 runs was executed after that was merged! We still have a lot of failing tests, but those are not flaky; they are failing (ref. comment above). cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" unstructuredv1 "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime/schema" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/discovery"

[x] test pr : kuberenetes/kubernetes# [x] two weeks soak start date : testgrid-link [x] two weeks soak end date : [x] test promotion pr : kubernetes/kubernetes#? According to this APIsnoop query, there are still some remaining APIService endpoints which are untested. With this query you can filter untested endpoints by their category and eligiblity for conformance. e.g below shows a query to find all conformance eligible untested,stable,core endpoints Addressing the following endpoints in this document; Using the current e2e test as a template, create an APIService for Using the REST client, read the APIService status (readApiregistrationV1APIServiceStatus) and confirm the value of . Using the REST client, patch the APIService (patchApiregistrationV1APIService) before confirming that it has been updated. Using the REST client, list all APIServices in the cluster (listApiregistrationV1APIService) before confirming that can be found. Clean up APIService resources used by the test. Due to the complexity of setting up the resources for APIService we have used the current e2e test as template. It has been extended in a for review. This query shows the endpoints hit within a short period of running the e2e test example endpoint | useragent ---------------------------------------|
createApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 deleteApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 listApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 patchApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 readApiregistrationV1APIService | Should be able to support the 1.17 Sample API Server using the current Aggregator2 readApiregistrationV1APIServiceStatus | Should be able to support the 1.17 Sample API Server using the current Aggregator2 (6 rows) Based on the initial APIsnoop query and the results listed above there are three new endpoints hit.

If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted


Question: Name: "example-int-port", }, Spec: v1.ServiceSpec{ <del> Selector: map[string]string{labelPod1: labelValue}, </del> <ins> Selector:                 map[string]string{labelPod1: labelValue}, PublishNotReadyAddresses: true, </ins> Ports: []v1.ServicePort{{ Name:       "example", Port:       80,

Which jobs are flaking: ci-kubernetes-kind-e2e-parallel pr:pull-kubernetes-e2e-gce pr:pull-kubernetes-e2e-gce-ubuntu-containerd pr:pull-kubernetes-e2e-gce-ubuntu-containerd ci-kubernetes-kind-ipv6-e2e-parallel Which test(s) are flaking: Second flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know: /sig network /assign
needs investigation since we're switching kube-proxy to endpointslice in this release
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
made me think of , but that's in already.
It's possible it's a test timeout that's just too close, but we need verification of that checking the controller and apiserver logs to make sure there isn't a latent bug
Thanks for catching this! It looks like the timeouts are actually associated with waiting for 2 pods to be ready, not actually anything related to EndpointSlice functionality - timeout is here: Given that, it seems like the obvious solution would be to update the Services to set to and update this block to wait for the Pods to have IPs, not for them to be ready. If that sounds reasonable, I'll work on getting a fix in tomorrow.
/remove-triage unresolved
If that is still exercises the same conditions (I vaguely remember publishNotReadyAddresses changing code paths in the endpoints controller... does it do the same in the endpointslice controller?), that seems reasonable to me.
I think the code path is very similar here, the only real difference should be how the EndpointSlice controller calculates readiness. I've gone ahead and filed a PR that uses this approach, hopefully it helps:
This flaked again this morning /reopen
Reopened this issue. <detailsIn response to : This flaked again this morning Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This should be /priority critical-urgent given
The top level error message does not make this clear but the issue here is that too many EndpointSlices are being created - 3 instead of 2.
Interesting... that condition should not persist for O(minutes), right?

that seems like it might be a bug, not just a test issue
There's potential that it's not consistently merging 2 ports with 1 endpoint into a single slice and instead creating a slice per port. It also could represent a larger bug. Trying to find a way to consistently reproduce this.
Despite my best efforts I have been unable to reproduce this on my own e2e cluster. I've had 10 test runs going at the same time repeatedly without any luck, any ideas for how to reproduce flaky test failures local would be welcome. In the meantime, I've created a follow up PR that will provide more helpful information if/when this test fails again: My leading theory right now is that this could be a caching issue. The EndpointSlice controller runs a sync, sees that we don't have an EndpointSlice for the given port combination in cache and then goes ahead and creates one, even though a matching EndpointSlice already exists and has not been cached yet. That would result in an annoying situation where an essentially duplicate EndpointSlice was created due to a stale cache. I would think the EndpointSliceTracker would resolve this issue for us though, so not sure. Getting the above PR in will help validate this theory. It's worth noting that if this is actually a duplicate EndpointSlice, clients like kube-proxy will handle this seamlessly because they already have to handle potential duplication in EndpointSlices. That would potentially explain why a test failure like this does not seem to be causing other things to break.
It appears that my theory was correct as witnessed here: I create a .

The key points I've noticed so far: 1 EndpointSlice references pod2 (example-named-port-f7vq5) 2 EndpointSlices reference pod1 (example-named-port-fkgcc and example-named-port-vfp2q) 2 EndpointSlices were created at "2020-07-28T04:57:51Z": example-named-port-f7vq5 (references pod2) example-named-port-vfp2q (references pod1) 1 EndpointSlice was created 1 second later ("2020-07-28T04:57:52Z") example-named-port-fkgcc (references pod1) All 3 EndpointSlices are: On generation 1 - no updates have occurred With a state of ready: true The 2 EndpointSlices referencing pod1 are effectively identical  Of course this is something the EndpointSliceTracker should be handling for us, but even that approach seems less than ideal here. It wouldn't actually prevent the duplicate EndpointSlices from being created, it should just be cleaning up duplicates after they happen. This combined with the race conditions referenced in make me think we may want to find a way to slow down the sync cycle for each Service here. This also feels at least tangentially related to the need for better rate limiting in both this and the Endpoints controller discussed in . /cc
Hmm - actually I don't understand why this is happening. Maybe I'm missing something, but I don't think slowing down controller is needed to solve this problem. Even though the controller usually is not lagging my more than 1s, it is possible that it can be lagging by a minute or more in pathological cases. So we can't just rely on the fact that it's guaranteed to be synced. Can you explain it a bit deeper why this is happening? i.e. some more detailed timeline like: T1: state of cache is XYZ T2: controller computes the ES x1, x2 T3: state of cache in ABC T4: controller computes ES y1 [why?]
Agree, I'm not in favor of slowing down the controller globally to address this. Delaying resync of a particular service until we observe the existence of new endpointslices we know we created (with some max delay) could be reasonable.

That's exactly where I was hoping to end up here, although I'm not quite sure how to do that cleanly yet. To be clear, this is not just to fix this flaky test. This flaky test is a symptom of a much larger and more annoying issue with the current implementation. The EndpointSlice controller is consistently running into errors where it's trying to update EndpointSlices that are stale. This is caused by follow up calls to running before the informer cache is up to date, which happens all the time during rolling updates where rapid churn of EndpointSlices is expected. Although this eventually resolves itself, it's causing a lot more load on API Server than necessary and can potentially result in some bad states where an update fails but a delete succeeds. In this specific instance, it looks like a syncService run is occurring after the first 2 EndpointSlices have been created but before the one referencing pod1 is in the informer cache. That results in the controller creating a new EndpointSlice to reference pod1 and thus we end up with a duplicate. If anything in the future triggers another syncService call that would be resolved. I'd initially thought that EndpointSliceTracker would be thing to trigger that follow up syncService, but I was wrong. That tracker just tracks the latest resource version for each EndpointSlice the controller has created/updated and compares that with EndpointSlices received through create/update events. If the controller doesn't think that EndpointSlice should exist or the controller's resource version for that slice does not match it triggers another syncService. In this case neither of those conditions would be met so no additional syncService call would be triggered. So that's a long way of saying that I think our best possible solution here is to introduce some delay between when Services can be synced. I think the easiest solution that would solve 99% of our problems would be to add something like a 1s delay in between syncs simply by delaying when was called. This would not be ideal, but it would result in an immediate improvement and be easy to backport. It's worth noting that although duplicate EndpointSlices like this or even attempts to update a stale EndpointSlice are not ideal, they are not actually invalid and will eventually resolve themselves. Consumers of this API already have to do some form of deduping.

With that said, I'm hopeful that there's a clean way to implement a solution that actually waited for each EndpointSlice change the controller has made to be reflected in the cache because that would be a better solution. I'll spend some time looking into possible implementations here.
I've created a draft PR that contains my initial progress towards the approach I described above. I don't particularly love this approach so I'm very open to other ways we could solve this issue. It's also still very much a WIP and has significant work remaining. I opened the PR early just so I could get some initial feedback on the direction and potentially inspire other ideas around approaches we could take here.
is still the best approach I can come up with here. It is ready for review if anyone has time to take a look.
Quick update here, I've limited the scope here to the following changes as discussed in Update the flaky test to consider duplicate EndpointSlices valid. Update the docs to make it very clear that consumers of the EndpointSlice API will have to do deduplication (as kube-proxy already does). Update the controller error handling to return immediately if a create, update, or delete errors (in that order). Reviews would be appreciated on all of the above.
Reopening since is not yet merged.
Thanks for catching that Closing now that is merged. /close
Closing this issue. <detailsIn response to : Thanks for catching that Closing now that is merged. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details