Question: return pvc, nil } <del> // Create a PV and PVC based on the passed in nfs-server ip and namespace. // There are 4 combinations, 3 of which are supported here: //   1) prebind and create pvc first //   2) no prebind and create pvc first //   3) no prebind and create pv first // The case of prebinding and creating the pv first is not possible due to using a // *generated* name in the pvc, and thus not knowing the claim's name until after // it has been created. // **Note: this function complements makePersistentVolume() and fills in the remaining // name field in the pv's ClaimRef. func createPVandPVC(c *client.Client, serverIP, ns string, pvFirst, preBind bool) (*api.PersistentVolume, *api.PersistentVolumeClaim, error) { </del> <ins> // Create a PVC followed by the PV based on the passed in nfs-server ip and // namespace. If the "preBind" bool is true then pre-bind the PV to the PVC // via the PV's ClaimRef. Return the pv and pvc to reflect the created objects. // Note: the pv and pvc are returned back to the It() caller so that the //   AfterEach func can delete these objects if they are not nil. // Note: in the pre-bind case the real PVC name, which is generated, is not //   known until after the PVC is instantiated. This is why the pvc is created //   before the pv. func createPVCPV(c *client.Client, serverIP, ns string, preBind bool) (*api.PersistentVolume, *api.PersistentVolumeClaim, error) { </ins> var bindTo *api.PersistentVolumeClaim <del> var err error pvc := makePersistentVolumeClaim(ns) // pvc.Name not known yet </del> <ins> var preBindMsg string </ins> <del> bindTo = nil if preBind { // implies pvc *must* be created before the pv pvFirst = false </del> <ins> // make the pvc definition first pvc := makePersistentVolumeClaim(ns) if preBind { preBindMsg = " pre-bound" </ins> bindTo = pvc } <ins> // make the pv spec </ins> pv := makePersistentVolume(serverIP, bindTo) <del> if pvFirst { By("Creating the PV followed by the PVC") pv, err = createPV(c, pv) } else { By("Creating the PVC followed by the PV") pvc, err = createPVC(c, ns, pvc) } </del> <ins> By(fmt.Sprintf("Creating a PVC followed by a%s PV", preBindMsg)) // instantiate the pvc pvc, err := createPVC(c, ns, pvc) </ins> if err != nil { return nil, nil, err } <del> if pvFirst { pvc, err = createPVC(c, ns, pvc) if err != nil { return pv, nil, err } } else { // need to fill-in claimRef with pvc.Name </del> <ins> // instantiate the pvc, handle pre-binding by ClaimRef if needed if preBind { </ins> pv.Spec.ClaimRef.Name = pvc.Name <del> pv, err = createPV(c, pv) if err != nil { return nil, pvc, err } </del> <ins> } pv, err = createPV(c, pv) if err != nil { return nil, pvc, err } return pv, pvc, nil } // Create a PV followed by the PVC based on the passed in nfs-server ip and // namespace. If the "preBind" bool is true then pre-bind the PVC to the PV // via the PVC's VolumeName. Return the pv and pvc to reflect the created // objects. // Note: the pv and pvc are returned back to the It() caller so that the //   AfterEach func can delete these objects if they are not nil. // Note: in the pre-bind case the real PV name, which is generated, is not //   known until after the PV is instantiated. This is why the pv is created //   before the pvc. func createPVPVC(c *client.Client, serverIP, ns string, preBind bool) (*api.PersistentVolume, *api.PersistentVolumeClaim, error) { preBindMsg := "" if preBind { preBindMsg = " pre-bound" } By(fmt.Sprintf("Creating a PV followed by a%s PVC", preBindMsg)) // make the pv and pvc definitions pv := makePersistentVolume(serverIP, nil) pvc := makePersistentVolumeClaim(ns) // instantiate the pv pv, err := createPV(c, pv) if err != nil { return nil, nil, err } // instantiate the pvc, handle pre-binding by VolumeName if needed if preBind { pvc.Spec.VolumeName = pv.Name } pvc, err = createPVC(c, ns, pvc) if err != nil { return pv, nil, err </ins> } return pv, pvc, nil

Context: NodeSwap feature gate was recently . Currently, as per , swap is only supported with cgroup v2 - cgroup v1 is no longer supported. For this reason, we decided to fail kubelet if NodeSwap feature gate is enabled and the node runs with cgroup v1. This would help prevent any surprises for the user by failing as soon as possible. This effort is being implemented in this PR: The problem: Currently, swap jobs are running with cgroup v1. Swap tests were in a way that they first check on which environment they're being run (e.g. which cgroup version? is NodeSwap FG enabled? Which swapBehavior is defined? etc) and then expect the right behavior to take place. With v1, the tests expect the containers to be defined without swap access. While the tests pass, they are much less valuable, since all they check is that swap is disabled for v1 workloads. The tests would be much more valuable if they would run with v2, therefore actually ensure that proper swap limitations are configured. In addition, is dependent on these changes, since as of now the jobs immediately fail with this PR since kubelet fails on an environment with cgroup v1 and swap enabled. What needs to change: [x] CRI-O swap related jobs should be defined to run with cgroup v2 (e.g. ) [x] containerd swap related jobs should be defined to run with cgroup v2 (e.g. ). Future work: In the future, we might consider introducing more jobs to check swap. Ideally, both and would be tested, perhaps by periodic jobs.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig testing
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: pods = filterHostPortConflicts(pods) ch := make(chan RunPodResult) <del> for _, pod := range pods { </del> <ins> for i := range pods { pod := pods[i] // Make a copy </ins> go func() { info, err := kl.runPod(pod) ch <- RunPodResult{&pod, info, err}

For example: struct field tag not compatible with reflect.StructTag.Get missing argument for Errorf("%v"): format reads arg 2, have only 1 args range variable pod enclosed by function struct field tag not compatible with reflect.StructTag.Get arg minion for printf verb %s of wrong type: unreachable code unreachable code
Those all look really important
I'm going to fix this unless I hear that someone else is working on this in the next couple minutes.


Question: self.mutex.Lock() defer self.mutex.Unlock() <del> // We cache both for efficiency and correctness if self.deviceMappings == nil { info, err := self.describeInstance() if err != nil { return "", false, err </del> <ins> info, err := self.describeInstance() if err != nil { return "", false, err } deviceMappings := map[mountDevice]string{} for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") { name = name[7:] </ins> } <del> deviceMappings := map[mountDevice]string{} for _, blockDevice := range info.BlockDeviceMappings { name := aws.StringValue(blockDevice.DeviceName) if strings.HasPrefix(name, "/dev/sd") { name = name[7:] } if strings.HasPrefix(name, "/dev/xvd") { name = name[8:] } if len(name) != 1 { glog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) } deviceMappings[mountDevice(name)] = aws.StringValue(blockDevice.Ebs.VolumeId) </del> <ins> if strings.HasPrefix(name, "/dev/xvd") { name = name[8:] } if len(name) != 1 { glog.Warningf("Unexpected EBS DeviceName: %q", aws.StringValue(blockDevice.DeviceName)) </ins> } <del> self.deviceMappings = deviceMappings </del> <ins> deviceMappings[mountDevice(name)] = aws.StringValue(blockDevice.Ebs.VolumeId) } for mountDevice, volume := range self.attaching { deviceMappings[mountDevice] = volume </ins> } // Check to see if this volume is already assigned a device on this machine <del> for mountDevice, mappingVolumeID := range self.deviceMappings { </del> <ins> for mountDevice, mappingVolumeID := range deviceMappings { </ins> if volumeID == mappingVolumeID { if assign { glog.Warningf("Got assignment call for already-assigned volume: %s@%s", mountDevice, mappingVolumeID)

Environment: CoreOS 815.0.0, k8s 1.1.0 () It happens to me quite often that a volume marked as "available" on AWS remains in a "detached" state according to the kubelet: Deleting the pod and recreating it eventually succeeds.
I have some fixes in progress that should fix this
/cc
I can reliably reproduce similar issue, I am not sure it's the same: create a AWS EBS, let's say it has ID Create a pod that uses this EBS: Create the pod. It succeeds eventually: Delete the pod Create the pod again. It will be forever IMO, it's caused by not removing the detached volume from internal map of attached volumes . Subsequent will find it there and it won't even try to attach it again. Patch is available in referenced PR.
can you please confirm that your bugs is the same as I can reproduce? Or do you see different bug, only with similar symptoms?
It is
Was this fixed by ?
I think you referenced the wrong issue / PR, didn't you? As far as I know this is not fixed yet.
Yes, I meant to link . mentioned there that it might fix this issue.
Oh right! It made it to v1.2.0-alpha.4
did not fix this issue (but fixed the DeviceMapping cache invalidation)
I think, what is happening is that you run into an error or timeout while waiting for the disk operation. That leaves the devicecache yet again in an inconsistent state. I have a fix here that just discards the cache on any error. Additionally, it is totally possible that there is concurrent operations which lead to an error and bork the cache. The fix: Unfortunately, this is in our tests still not 100% rock solid. Especially in timeout cases. I'm missing the capability to properly test this to engage in a bigger refactoring of this whole issue. A workaround to actually fixing it properly would be to add a TTL to the cache, so that it eventually starts working again.
Just for information for people who might use the GitHub search, as of Kubernetes 1.1.7 the errors logged by the look like: Deleting the pod or restarting doesn't help, the instance keeps failing attaching the EBS with the same errors if the pod get rescheduled on the same host.

thanks for the fix!
Oh... the same host. I bet I messed up that case... I'm going to move into 1.2 so I take another look at this!
What's the status on this?
I just don't think this is fixable with the volume cache we have. I proposed a PR to remove the cache, and replace it with a map of active attachment operations in progress. A little less efficient, but a lot more correct (presuming passes testing).
How to check what K8 version has this fix?
Is there any progress on this issue? I'm running K8s version v1.5.2, and I'm having similar issues. awsElasticBlockStore volumes are virtually unusable due to one of three errors I'm getting: My volume claim is ReadWriteOnce, so this next error confuses me: And finally: [SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected., SchedulerPredicates failed due to persistentvolumeclaims "mysql" not found, which is unexpected.] Unable to mount volumes for pod "staging-connect-mysql-<id(<id)": timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql] Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"<id". list of unattached/unmounted volumes=[mysql]
Also having the same issue as ^ Should we be using PV+VPC instead of claiming the EBS volume directly from the deployment spec?


Question: return nil, err } <del> return &GenericWebhook{restClient, initialBackoff}, nil </del> <ins> return &GenericWebhook{restClient, initialBackoff, DefaultShouldRetry}, nil </ins> } // WithExponentialBackoff will retry webhookFn() up to 5 times with exponentially increasing backoff when <del> // it returns an error for which apierrors.SuggestsClientDelay() or apierrors.IsInternalError() returns true. </del> <ins> // it returns an error for which this GenericWebhook's ShouldRetry function returns true, confirming it to // be retriable. If no ShouldRetry has been defined for the webhook, then the default one is used (DefaultShouldRetry). </ins> func (g *GenericWebhook) WithExponentialBackoff(ctx context.Context, webhookFn func() rest.Result) rest.Result { var result rest.Result <ins> shouldRetry := g.ShouldRetry if shouldRetry == nil { shouldRetry = DefaultShouldRetry } </ins> WithExponentialBackoff(ctx, g.InitialBackoff, func() error { result = webhookFn() return result.Error() <del> }) </del> <ins> }, shouldRetry) </ins> return result } <del> // WithExponentialBackoff will retry webhookFn() up to 5 times with exponentially increasing backoff when // it returns an error for which apierrors.SuggestsClientDelay() or apierrors.IsInternalError() returns true. func WithExponentialBackoff(ctx context.Context, initialBackoff time.Duration, webhookFn func() error) error { </del> <ins> // WithExponentialBackoff will retry webhookFn up to 5 times with exponentially increasing backoff when // it returns an error for which shouldRetry returns true, confirming it to be retriable. func WithExponentialBackoff(ctx context.Context, initialBackoff time.Duration, webhookFn func() error, shouldRetry func(error) bool) error { </ins> backoff := wait.Backoff{ Duration: initialBackoff, Factor:   1.5,

What would you like to be : The api server should retry sending audit events to configured webhook sinks regardless of the type of HTTP error code it might receive from the webhooks. Why is this needed: While some components may live with failed data transmissions, this is a particularly sensitive topic for the audit area. The API server audit mechanisms must ensure that no other reasonable option exists before dropping an audit event that may be the only trace of malicious behavior in the cluster. It seems that the current implementation may be improved further in this direction. Currently, when the API server is configured for non-blocking sending of audit events to a webhook audit sink backend, it will retry up to 5 times with exponential backoff, should there be an error in communicating with the webhook, as implemented . However, not all errors will trigger a retry. Some are considered transient and yield retry. Others simply stop the retry cycle. For example, it retries on receiving HTTP code , but not on receiving . The list of error codes that will yield a retry is actually quite small - three HTTP errors and one network error. Any other error will not trigger a retry and the audit events will be lost. The code comments explain that the retry will consider only recoverable failures, which may make sense in other cases, but as far as sending audit events to an audit backend webhook is concerned, it is not sufficient policy. A more robust approach would be to consider any error and give the backend service a chance to recover instead of relying on the error codes semantics and the webhook implementations interpretation of it. That is more inline with having the transmission of audit events as top priority. /sig auth /sig api-machinery


Question: return expClient.Deployments(ns).DeleteCollection(nil, api.ListOptions{}) } <ins> func deleteReplicaSets(expClient extensions_unversioned.ExtensionsInterface, ns string) error { return expClient.ReplicaSets(ns).DeleteCollection(nil, api.ListOptions{}) } </ins> func deleteIngress(expClient extensions_unversioned.ExtensionsInterface, ns string) error { return expClient.Ingresses(ns).DeleteCollection(nil, api.ListOptions{}) }

The delete command for a replicaset hangs forever. This then manifests as phantom RSs when I delete and recreate a namespace.
- delete is the gift that keeps on giving
cc
Had you already deleted the corresponding Deployment?
I am still seeing this problem. Deployments and replica sets are sticking.
From what I can tell, calling DELETE_COLLECTION on Deployments is returning: And since the controller assumes not found errors are equivalent to already deleted, it thinks the content is deleted. When I manually inspect, I still see the deployment and replica set laying around... //cc
holding on this while i debug some more...
This was a false alarm, apologies.


Question: "Time": recent(maxStatsAge), // We don't limit system container memory. "AvailableBytes":  BeNil(), <del> "UsageBytes":      bounded(1*framework.Mb, 10*framework.Gb), "WorkingSetBytes": bounded(1*framework.Mb, 10*framework.Gb), </del> <ins> "UsageBytes":      bounded(1*framework.Mb, memoryLimit), "WorkingSetBytes": bounded(1*framework.Mb, memoryLimit), </ins> // this now returns /sys/fs/cgroup/memory.stat total_rss <del> "RSSBytes":        bounded(1*framework.Mb, 1*framework.Gb), </del> <ins> "RSSBytes":        bounded(1*framework.Mb, memoryLimit), </ins> "PageFaults":      bounded(1000, 1E9), "MajorPageFaults": bounded(0, 100000), }),

/kind bug What happened: e2e node summary test fails on machines with more than 10Gb of ram. What you expected to happen: memory limit bounds to look at node capacity rather than hardcode an upper bound. How to reproduce it (as minimally and precisely as possible): have more ram on a box and run e2e-node FOCUS=Summary Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/api/testapi" "k8s.io/kubernetes/pkg/api/v1" <del> "k8s.io/kubernetes/pkg/apis/extensions" </del> <ins> extensions "k8s.io/kubernetes/pkg/apis/extensions/v1beta1" </ins> ) func newPod(podName string, nodeName string, label map[string]string) *v1.Pod {

Here are the hash label keys of all controllers: Deployment: DaemonSet: (new in 1.7) StatefulSet: (new in 1.7) Had a brief discussion with and about this, it should be consistent. We can use for DaemonSet and StatefulSet, and keep Deployment as-is. Deployment hash label key should be different from other two because Deployment pods hash labels is used by selector (unlike the other two controllers). We probably should have used hash annotations instead of hash labels for DaemonSet and StatefulSet, since we don't use hash labels to select their history or pods. Hash labels are chosen for consistency. This is fine since we don't modify hash label values.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Actually, controller revisions created by statefulset, daemonset has different label: // controller revision created by statefulset label: "" // controller revision created by daemonset label: controller-revision-hash: "" should be consistent?


Question: funcOut = file } <del> docsForTypes := pkg_runtime.ParseDocumentationFrom(*typeSrc) </del> <ins> docsForTypes := kruntime.ParseDocumentationFrom(*typeSrc) </ins> if *verify == true { <del> rc, err := pkg_runtime.VerifySwaggerDocsExist(docsForTypes, funcOut) </del> <ins> rc, err := kruntime.VerifySwaggerDocsExist(docsForTypes, funcOut) </ins> if err != nil { fmt.Fprintf(os.Stderr, "Error in verification process: %sn", err) }

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: Requests: podRes, Limits:   podRes, }, <del> }) </del> <ins> }, framework.PodStartShortTimeout) </ins> defer func() { // Clean-up the critical pod

Which jobs are flaking: sig-release-master-blocking Conformance - GCE - master Which test(s) are flaking: E2E: [sig-scheduling] SchedulerPreemption [Serial] validates basic preemption works [Conformance] Testgrid link: Reason for failure: Anything else we need to know: is currently unavailable links to specific failures in spyglass appreciated From TestGrid alert ...
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
I cannot spot any significant logic flaw in the test. A possible cause is the timeout is set too short (1 minute), which may be not enough for the terminating pod to run. Notice that unlike integration test, the time includes the end to end cycle: (1) preemptor being scheduled and evict low-priority pods, (2) low-priority pods being terminating -reported by kubelet that the victims have been deleted, (3) scheduler gets notifications from API server that the victims have been deleted, (4) preemptor gets scheduled, (5) kubelet reports the preemptor is running. /cc for more thoughts.
Maybe, some logs to / would help. Just making the timeout longer don't make sense.
Is addressing this flake?
I think this flake happens before the failure tries to fix:
/kind flake
That's indeed a lot of steps, and flakiness seems very low. I think increasing the timeout for preemption seems reasonable.
is this flake a must-fix in 1.20?


Question: # See the OWNERS docs at https://go.k8s.io/owners <ins> # Disable inheritance as stable metrics require SIG Instrumentation approval options: no_parent_owners: true </ins> approvers: - sig-instrumentation-approvers reviewers: - sig-instrumentation-reviewers labels: - sig/instrumentation <ins> - area/stable-metrics </ins>

Currently, stable metric changes can be approved by top-level approvers and are not obviously "special" to a contributor. We have verify scripts that will prevent adding or changing stable metrics without updating Instrumentation-owned files, but the errors can easily be "fixed" by running the corresponding update script. Contributors may end up doing so to fix the verify errors and not understand the consequences of promoting a metric to "stable" status. We would like to consider the following: [x] Migrate the documentation for stable metrics from KEP to the community/developer docs: [x] Add a new label that flags stable metrics changes, ex. , from the OWNERS file in required for stable metrics [x] When the label is applied, a comment from the bot that alerts contributors of the stable metric guidelines, similar to [ ] Do we also want to consider adding to the stable metrics OWNERS file to avoid accidental merges from top-level approvers? We had some contributors join the latest SIG Instrumentation meeting requesting to add some labels to stable metrics. They were under the impression that stable metrics worked similar to other Kubernetes APIs, and thus they could make "compatible" API changes. This is not the case, and they were unaware of the stable metrics documentation. I think it would be good to flag this documentation to anyone making stable metrics changes, in the same way that we flag API changes and point contributors to that documentation. /sig instrumentation contributor-experience /cc
I don't think we had any issue with top-level approvers graduating metrics to stable in the past, but I also don't think they should be able to do that, so I would be in favor of that change.
Related, stable metrics changes should some day trigger a docs PR to update a reference doc. (there's no reference doc listing stable metrics yet, but I hope that changes)
/triage accepted
/assign ehashman


Question: framework.ExpectNoError(err, "Unable to re-create the deleted node") framework.ExpectNoError(framework.WaitForGroupSize(framework.TestContext.CloudConfig.NodeInstanceGroup, int32(initialGroupSize)), "Unable to get the node group back to the original size") framework.WaitForNodeToBeReady(f.ClientSet, nodeToDelete.Name, nodeStatusTimeout) <ins> framework.WaitForAllNodesSchedulable(f.ClientSet, nodeStatusTimeout) </ins> nodes = framework.GetReadySchedulableNodesOrDie(f.ClientSet) <del> if len(nodes.Items) != originalCount { return fmt.Errorf("The node count is not back to original count") } return nil </del> <ins> Expect(len(nodes.Items)).To(Equal(originalCount), "Requires node count to return to original node count.") </ins> }() By("submitting host0Pod to kubernetes")

Since 2 days ago, the test is becoming more and more flaky, and one of the examples is: https://k8s-
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
please triage this one. Currently I put it as the release blocker, and please feel free to remove that if you think it is not. Thanks!
The test deletes the node, and at the end, waits 5 minutes for the node to become ready, but it timed out. cc
The test logic hasn't changed after I last modified it, so the test deletes the node through gcloud, and the instance should be restarted automatically by GCE because it's part of an instance template. The likely possibility, as pointed out, is that another change made node discovery take longer, so extending the timeout would be enough. The very unlikely possibility is there's a problem discovering brand new instances.


Question: server := &http.Server{ Addr:           net.JoinHostPort(address.String(), strconv.FormatUint(uint64(port), 10)), Handler:        &s, <del> ReadTimeout:    5 * time.Minute, WriteTimeout:   5 * time.Minute, </del> MaxHeaderBytes: 1 << 20, } glog.Fatal(server.ListenAndServe())

I noticed that if I'm following a pod log that is being actively written to it will timeout after 5 minutes. However if the pod is not writing logs the request will not timeout. In that case the request will remain open seemingly indefinitely. I can change the behavior by modifying this line of code: To make it happen faster you can set the value lower. I've been testing with 1 minute. Steps to reproduce: cluster the nginx pod in another terminal generate log output by hitting the pod: (you'll need to change the ip). the logs: In this case whenever the WriteTimeout is reached curl will receive the zero sized chunk and the connection will be closed. If you leave out step 3 curl will continue seemingly indefinitely. This behavior definitely seems odd. Is the usage of WriteTimeout correct here? A quick search shows that others have been confused by the intention of WriteTimeout which is why I ask.
is this same as ?
From discussions on the OpenShift side it sounds somewhat related however what we're talking about is a reasonable (but arbitrary) timeout set. Users of are subjected to this timeout with no good way to pick up where they left off to avoid redownloading potentially large quantities of logs.
To me, it would make sense if the api endpoint had argument which would fetch logs from line (it's easy to track on client on which line we're at), exactly what does: I'm assuming that kubelet gets logs using docker api, unfortunately there is no such thing in docker's api:
I think users that want to do this should get the logs from elastic search instead of from kubectl.
I can't say I disagree about needing something like elastic search in a real world deployment. I guess part of the weirdness is that clients of the logging API rarely know how much is going to be pulled down before they make the request. I guess the answer now if you're following the logs is "5 minutes worth". I know this isn't how it's implemented today but in my mind following the logs should behave more like or . Meaning, when you state you want to follow you don't fetch the entire log file you just get wherever it happens to be (and maybe a little more for context). If you want the whole file you don't pass in .
Any update? This is really annoying and I'm thinking of patching my k8s.

I think the default API timeout has been lengthened to 60 minutes Defined: Used for API server: It looks like the kubelet timeout is still 5 minutes, so that would gate log streaming to 5 minutes:
More nuances in Looks like the API http server times out after 60 minutes: Watches time out between 5 and 10 minutes: Kubelet http server still times out after 5 minutes
Which means that getting logs timeouts after 5 minutes?
yeah
did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
nm, just merged the pr. did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
we could easily bump up the kubelet timeout. The handlers on the kubelet don't have their own built in timeouts (like I did for watch), so a potential downside is long hanging /stats requests could hog connections, and depending on why they're hanging, continue consuming kubelet/cpu/cadvisor time. had a suggestion previously about using elasticsearch, are you running with a cluster that doesn't have the addons (, if you set it up with kube-up it should)?
IMO there's no reason not to make kubelet server's timeout really long, at least for these requests.
I'm trying to put in a fix to increase the kubelet timeouts to 1 hour to match the master. Unfortunately adding proper timeout handling for these cases may not make 1.0, but hopefully this can make it less annoying.


Question: if !reflect.DeepEqual(n.podCIDRs, podCIDRs) { klog.ErrorS(nil, "Using NodeCIDR LocalDetector mode, current PodCIDRs are different than previous PodCIDRs, restarting", "node", klog.KObj(node), "newPodCIDRs", podCIDRs, "oldPodCIDRs", n.podCIDRs) <del> panic("Current Node PodCIDRs are different than previous PodCIDRs, restarting") </del> <ins> klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> } }

There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. As suggested by in , we can replace by /sig network /help /kind cleanup
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : There are some cases when kube-proxy has to exit and restart, in order to do that we panic, but this has the problem that the log output shows a literal "panic" message and engineers, support, customers and automation gets confused, thinking that is a problem when is the expected behavior. /help /kind cleanup </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to raise a PR to fix this.


Question: # Restart the kube-apiserver on a node ($1) function restart-apiserver { <del> ssh-to-node "$1" "sudo /etc/init.d/kube-apiserver restart" </del> <ins> ssh-to-node "$1" "sudo docker kill `sudo docker ps | grep /kube-apiserver | awk '{print $1}'`" </ins> } # Perform preparations required to run e2e tests

Now that kube-apiserver runs in a container, no longer exists: /cc
Actually fixed by .


Question: func sshRestartMaster() error { if !ProviderIs("gce", "aws") { <del> return fmt.Errorf("unsupported provider: %s", TestContext.Provider) </del> <ins> return fmt.Errorf("unsupported provider for sshRestartMaster: %s", TestContext.Provider) </ins> } var command string if ProviderIs("gce") {

is failing with "framework.RestartControllerManager() = unsupported provider: gke; want nil" /kind failing-test
/sig network
Introduced in , should be fixed by


Question: nodeAllocatableMilliCPU2, nodeAvailableMilliCPU2 := getNodeAllocatableAndAvailableMilliCPUValues(&node) framework.Logf("TEST2: Node '%s': NodeAllocatable MilliCPUs = %dm. MilliCPUs currently available to allocate = %dm.", node.Name, nodeAllocatableMilliCPU2, nodeAvailableMilliCPU2) <del> testPod3CPUQuantity := resource.NewMilliQuantity(nodeAvailableMilliCPU2+testPod1CPUQuantity.MilliValue()/2, resource.DecimalSI) </del> <ins> testPod3CPUQuantity := resource.NewMilliQuantity(nodeAvailableMilliCPU2+testPod1CPUQuantity.MilliValue()/4, resource.DecimalSI) </ins> testPod1CPUQuantityResized := resource.NewMilliQuantity(testPod1CPUQuantity.MilliValue()/3, resource.DecimalSI) framework.Logf("TEST2: testPod1 MilliCPUs after resize '%dm'", testPod1CPUQuantityResized.MilliValue())

/kind failing-test /sig node
/assign
/triage accepted
Seems to me that problem is related with test-infra config ? I noticed Checking kubelet logs there are complains about kubelet not finding configuration file. Even if this job run in AWS it is defined under ( ) and not under aws subfolder. Performing vimdiff periodic- ./aws/periodic-eks- i notice some variables are missing, but i don't have permissions to re-run the job to be able to debug more on this. Test code for InPlacePodVerticalScaling doesn't seem to be the root cause here, if we could increase the kubelet log level and perform a rerun might be helpful to debug more. Checking containerd logs , it seems that a container failed to stop, this might explain why pod resize failed due to lack of resources)
These two tests are permanent failing on this job
True, but looking deeper it seems for the first test it fails ( unable to resize there are no available resources ) because a previous pod has not been deleted according to containerd logs . For second error it seems to be a config error , it complains that it could not login. I believe something is wrong with the setup ( repeated kubelet config errors in kubelet logs )
/sig storage
More information for the container delete error, it belongs to mainpod-with-poststart-http-hookrestartable-init-container-lifecycle-hook. I will try to check if InPlacePodVerticalScaling test can be improved to cover this corner case. Please note, Garbage collect error happens before the InPlacePodVerticalScaling test failure, perhaps this is the root cause of the failure in InPlacePodVerticalScaling error. Perhaps we should update Issue title and include the garbage collect error for this issue? ! Source: / . Please search for


Question: epIpt := netlinktest.ExpectedIptablesChain{ string(kubeServicesChain): {{ JumpChain: string(KubeLoadBalancerChain), MatchSet: kubeLoadBalancerSet, <ins> }, { JumpChain: string(KubeMarkMasqChain), MatchSet: kubeClusterIPSet, }, { JumpChain: string(KubeNodePortChain), MatchSet: "", }, { JumpChain: "ACCEPT", MatchSet: kubeClusterIPSet, }, { JumpChain: "ACCEPT", MatchSet: kubeLoadBalancerSet, </ins> }}, string(KubeLoadBalancerChain): {{ JumpChain: string(KubeFireWallChain), MatchSet: kubeLoadbalancerFWSet,

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --ipvs mode didn't preserve client IP for NodePort Service with protocol SCTP and ExternalTrafficPolicy Local. iptables mode works fine. ipvs mode with TCP and UDP protocol works fine. Client IP for NodePort Service should be preseved when ExternalTrafficPolicy is Local regardless of the mode and the protocol. kube-proxy mode to ipvs a NodePort service with protocol SCTP and ExternalTrafficPolicy Local the NodePort service, the client IP was not preserved This is caused by wrong order of iptables rules that match kubeNodePortLocalSetSCTP and kubeNodePortSetSCTP: Kubernetes version (use ): v1.22.0 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network /assign
/triage accepted


Question: { name: "A pod has a toleration that keys and values match the taint on the node, but (non-empty) effect doesn't match, " + "can't be scheduled onto the node", <del> pod:        podWithTolerations("pod1", []v1.Toleration{{Key: "foo", Operator: "Equal", Value: "bar", Effect: "PreferNoSchedule"}}), node:       nodeWithTaints("nodeA", []v1.Taint{{Key: "foo", Value: "bar", Effect: "NoSchedule"}}), wantStatus: unschedulable, </del> <ins> pod:  podWithTolerations("pod1", []v1.Toleration{{Key: "foo", Operator: "Equal", Value: "bar", Effect: "PreferNoSchedule"}}), node: nodeWithTaints("nodeA", []v1.Taint{{Key: "foo", Value: "bar", Effect: "NoSchedule"}}), wantStatus: framework.NewStatus(framework.UnschedulableAndUnresolvable, "node(s) had taint {foo: bar}, that the pod didn't tolerate"), </ins> }, { name: "The pod has a toleration that keys and values match the taint on the node, the effect of toleration is empty, " +

When trying to see why pods don't start using either kubectl get events or kubectl describe pod , the base message is not detailed enough. Warning FailedScheduling 5s (x4 over 65s) default-scheduler 0/1 nodes are available: 1 node(s) had taints that the pod didn't tolerate. What would you like to be : Details of the taint and details of what it didn't tolerate Why is this needed: To get much much better debug information by default when trying to understand why pods can't launch.
/sig cli
/sig scheduling /priority important-longterm
/cc
/remove-sig cli
Those labels are not set on the issue: <detailsIn response to : /remove-sig cli </details
/good-first-issue
/assign starizard
Can I get an idea of what this would involve? I might be able to put some work on it.
I assume is taking this already. The error message is produced in We probably also require changes in to get the name of the missing toleration.
/remove-help


Question: } var wg sync.WaitGroup wg.Add(1) <del> go func(volSpec *volume.Spec, expectAttach bool) { attachID, err := csiAttacher.Attach(volSpec, types.NodeName("fakeNode")) </del> <ins> go func(volSpec *volume.Spec) { attachID, err := csiAttacher.Attach(volSpec, "fakeNode") </ins> defer wg.Done() if err != nil {

Which test(s) are flaking: TestAttacherWithCSIDriver Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: a test binary for the affected package: the affected test using the tool (setting the timeout to a generous timeout relative to the normal test runtime of .02 seconds): failures are seen, look at the logs to see the details about the error: /sig storage cc
/assign can you help investigate this flake?
I have found the root cause. /assign
filed to address this issue.


Question: discoveryv1 "k8s.io/api/discovery/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" <ins> genericapiserver "k8s.io/apiserver/pkg/server" </ins> "k8s.io/client-go/kubernetes/fake" <ins> corev1client "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/rest" </ins> core "k8s.io/client-go/testing" "k8s.io/kubernetes/pkg/controlplane/reconcilers" <ins> corerest "k8s.io/kubernetes/pkg/registry/core/rest" </ins> netutils "k8s.io/utils/net" )

Clusters created with kubeadm using DualStack are not able to work, because CoreDNS pods never get ready Seeing one failed job logs It seems that the pod is "waiting for the kubernetes service"?? coredns version seems the same as in the working jobs CoreDNS pod should be ready Instal a Dual-stack cluster using master with kubeadm (or kind ) ! It started to fail on Oct12 <details</details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc I didn't notice it was failing for more than 1 month :facewithhead_bandage: I didn't have time to dig into it yet, but this PR landed just that day how is that the pods running has the version 1.8.4 but the PR seems to update kubeadm to 1.8.6?
Looking into this today. The error from CoreDNS should only happen when the kubernetes plugin fails to sync with the Kubernetes API, either because it can’t connect, or it can’t sync the lists of services/endpointslices. There were changes and fixes to the API connection code, so it could be a regression.
Actually no. The changes I was thinking of were already in 1.8.4.
How can we reproduce it : Instal a Dual-stack cluster using master with kubeadm (or kind ) I was unable to reproduce this issue in a kind dual stack cluster.

After upgrading coredns to 1.8.6 (from 1.8.0 installed by kind), and adding list/watch permission for endpointslice, the pods were ready. edit: It also works for coredns 1.8.4 - I'm unclear as to which version I should be checking.
let me check if is a regression in kind :thinking:
working with 1.8.6 and yesterdays kubernetes podIPs
This is weird, I hit a similar issue in openshift, endpoints doesn't match the IP family of the kubernetes.default service apiserver service is single stack, at least until this KEP fixes it What I don't understand is why the order of the IPs in kind is different let me try to fix both things /assign
my fault, sorry for the noise :sweat_smile:


Question: } itArg := testArg It(name, func() { <ins> uuid := string(util.NewUUID()) </ins> totalPods := itArg.podsPerMinion * minionCount <del> RCName = "my-hostname-density" + strconv.Itoa(totalPods) + "-" + string(util.NewUUID()) </del> <ins> nameStr := strconv.Itoa(totalPods) + "-" + uuid ns = "e2e-density" + nameStr RCName = "my-hostname-density" + nameStr // Create a listener for events events := make([](*api.Event), 0) _, controller := framework.NewInformer( &cache.ListWatch{ ListFunc: func() (runtime.Object, error) { return c.Events(ns).List(labels.Everything(), fields.Everything()) }, WatchFunc: func(rv string) (watch.Interface, error) { return c.Events(ns).Watch(labels.Everything(), fields.Everything(), rv) }, }, &api.Event{}, time.Second*10, framework.ResourceEventHandlerFuncs{ AddFunc: func(obj interface{}) { events = append(events, obj.(*api.Event)) }, }, ) stop := make(chan struct{}) go controller.Run(stop) // Start the replication controller </ins> RunRC(c, RCName, ns, "gcr.io/google_containers/pause:go", totalPods) <ins> By("Waiting for all events to be recorded") last := -1 current := len(events) timeout := 10 * time.Minute for start := time.Now(); last < current && time.Since(start) < timeout; time.Sleep(10 * time.Second) { last = current current = len(events) } close(stop) if current != last { Logf("Warning: Not all events were recorded after waiting %.2f minutes", timeout.Minutes()) } Logf("Found %d events", current) // Verify there were no pod killings or failures By("Verifying there were no pod killings or failures") for _, e := range events { for _, s := range []string{"kill", "fail"} { Expect(e.Reason).NotTo(ContainSubstring(s), "event:' %s', reason: '%s', message: '%s', field path: '%s'", e, e.ObjectMeta.Name, e.Message, e.InvolvedObject.FieldPath) } } </ins> }) }

Currently the density tests are ok if all the pods come up eventually. We should verify there are no failure/error events while the pods are coming up. This could indicate issues dealing with the number of pods and should be investigated.
has been merged, and there's already an issue filed for followup actions, so I think we can close this.


Question: pathOptions.EnvVar = "" streams, _, buf, _ := genericclioptions.NewTestIOStreams() cmd := NewCmdConfigView(cmdutil.NewFactory(genericclioptions.NewTestConfigFlags()), streams, pathOptions) <ins> // "context" is a global flag, inherited from base kubectl command in the real world cmd.Flags().String("context", "", "The name of the kubeconfig context to use") </ins> cmd.Flags().Parse(test.flags) if err := cmd.Execute(); err != nil { t.Fatalf("unexpected error executing command: %v,kubectl config view flags: %v", err, test.flags) }

/kind bug What happened: Current context was set to cluster1. Ran and got back information about cluster1 instead. What you expected to happen: I expected the output to be the cluster2 information How to reproduce it (as minimally and precisely as possible): Have two contexts in your kube config, A and B Run to set A as current context Run Anything else we need to know?: I'd like to help contribute, so if this is an easy fix, someone point me in the right direction and I'll try to get it done. Environment: Kubernetes version (use ): 1.10.2 OS (e.g. from /etc/os-release): MacOS 10.11.6 Install tools: kubectl was installed via brew
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Filed


Question: } } } <ins> func TestConvertSliceStringToDeletionPropagation(t *testing.T) { tcs := []struct { Input  []string Output v1.DeletionPropagation }{ { Input:  nil, Output: "", }, { Input:  []string{}, Output: "", }, { Input:  []string{"foo"}, Output: "foo", }, { Input:  []string{"bar", "foo"}, Output: "bar", }, } for _, tc := range tcs { var dp v1.DeletionPropagation if err := v1.Convert_Slice_string_To_v1_DeletionPropagation(&tc.Input, &dp, nil); err != nil { t.Errorf("Convert_Slice_string_To_v1_DeletionPropagation(%#v): %v", tc.Input, err) continue } if !apiequality.Semantic.DeepEqual(dp, tc.Output) { t.Errorf("slice string to DeletionPropagation conversion failed: got %v; want %v", dp, tc.Output) } } } </ins>

makes API server accepts DeleteOptions as a query parameter, but APIserver cannot parse , which is a new field in 1.6. Probably it's a bug in the parameterCodec. cc
IMO this is not a 1.6 release blocker
does this apply to all DeleteOptions fields? does a 1.5 server handle the orphanDependents field sent by correctly?
No, is handled correctly by the apiserver built from head. And for , because our generated clients still encode DeleteOptions as body, so they are not affected.
- we can close this one right?
/sig api-machinery
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Why was this closed? I'm using a Kubernetes 1.9 cluster and this is still an issue. If I pass propagationPolicy as part of the query string (as per the documentation), I still receive the same error message. If I pass a DeleteOptions object with the propagationPolicy within, it works.
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
what documentation were you referring to? Is it the openapi doc? I don't know if we can just deprecate the support of query parameters for DELETE. I understand it's a breaking change, but it has been broken for a year.
Everything in delete needs to be moved to query parameters (unfortunately) as it is apparently non-standard to have a body in a delete request.

Chao Xu <:
/reopen
/remove-lifecycle rotten
cc
/assign
The DELETE operation states that propagationPolicy is a query parameter. Is correct that it's non-standard to have a body on a delete request.
GKE default version is still 1.9.7


Question: By("creating a third pod to probe DNS") pod3 := createDNSPod(f.Namespace.Name, wheezyProbeCmd, jessieProbeCmd, true) <del> validateTargetedProbeOutput(f, pod3, []string{wheezyFileName, jessieFileName}, "10.0.0.123") </del> <ins> svc, err := f.ClientSet.Core().Services(f.Namespace.Name).Get(externalNameService.Name, metav1.GetOptions{}) Expect(err).NotTo(HaveOccurred()) validateTargetedProbeOutput(f, pod3, []string{wheezyFileName, jessieFileName}, svc.Spec.ClusterIP) </ins> }) })

This e2e test is failing on several of the upgrade tests. https://k8s- https://k8s- https://k8s- /cc
The upgrade tests you listed mean running 1.6/1.7 tests against master clusters, right? was merged after 1.7. Before that PR, clusterIP is not respected when updating from service type ExternalName to type ClusterIP. After that PR, specified clusterIP will be allocated. The IP will be validated when allocating. In the failed tests, is not in the valid clusterIP range of the testing clusters. Maybe we should disable this test case in such upgrade tests? /cc
how to disable a test in upgrade tests?
tried to check with #sig-release or any advice on this?
would it be possible to modify the test to make it pick a valid clusterIP range?
On HEAD, it already picks a valid clusterIP, but I don't how to modify in 1.7/1.6. Maybe by a cherry-pick but I don't know if we want to cherry-pick .
If you can identify the patch that fixes the bug, we could backport. Or we should disable that case.
. But it's somewhat large, I don't think we want to backport it. One alternative is to send a small patch to change below IP to be another valid IP (although it's already a valid IP on HEAD). So we can have a small patch to backport and then we can cherry-pick it to 1.6/1.7. Tim, WDYT?
This test started passing again on the upgrade tests. hmmm. https://k8s-
Hmmm, weird...
Can't find anything in the log explaining why it is not failing anymore. Only thing I can see from comparing two logs artifacts (, ) is the name of sub directories (that I guess is related to notes running the test) is different (and less in success case). Maybe it has something to do with some config changes in running e2e tests.
also comparing image list of files of those two , cases, the success case has way less images (only one actually). I wonder if some change disabled this (or many more) tests?!
Any update on this?

The test is still failing for at least gce-1.7-master-upgrade-master. Should the test being disabled for 1.7 and re-enabled in 1.8?
I take a look at the test failure of . There is only one failure of (the test case associated with this issue) on 08.29 since 08.17. The is as below. It's not the failure in this issue.
As is mentioned in by , the original issue here might be caused by upgrade framework.
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/network: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
is this still regularly failing? And is the actual issue the upgrade framework, or something actually DNS related?
Going to close since the test is back to green. Will re-open if we see more failures.
And is the actual issue the upgrade framework, or something actually DNS related? Per discussion in , I think it's the issue of upgrade framework.
This is popping up again in some of our other upgrade tests https://k8s- Going to re-open for tracking. It sounds like this is still non-blocking though, so lowering the priority.
/sig testing /area test-infra /assign as per
Found this in : https://k8s- Not sure if this has something to do with the failure. BTW, the below tests have exactly the same failure status when fails. [] CronJob should adopt Jobs it owns that don't have ControllerRef yet [] Initializers should be invisible to controllers by default [] PodPreset should create a pod preset [] PodPreset should not modify the pod on conflict So I think it's most likely an issue of the upgrade framework.
/unassign That's the normal error message from a Ginkgo test failure and has nothing to do with , which has the failure signature of .
/assign xiangpengzhao
I'll track you guys as owners for this failure. Let's try to get it merged quickly.

And let me know if you don't have time/power to fix that. Thanks
Upgrade framework is likely broke by Ref: It is being fixed by
ref:
- does not seem to have v1.8 and this Issue seems to have v1.8, is there a disconnect? ( cc )
I want to be in 1.8 milestone and be cherry-picked to 1.6/1.7.
I it to the 1.8 milestone, but it still needs approvals. Contact about the 1.7.x cherry pick and about 1.6.x
Thanks !
merged, let's wait and see :)
https://k8s- is green!
Unfortunately, gke-gci-1.7-gci-master-upgrade-cluster-parallel still fails after merged. This test has a different clusterIP range . cc https://k8s- PS: helps provide the range info, it's really useful :)
I send PR trying to fix this. PTAL. Thanks!
This is still failing on some 1.8 upgrade tests https://k8s-
do we need to cherrypick the fix into previous branches?
Sorry I forgot to update here. is fixing the gke 1.7-1.8 upgrade cases. I will send a similar fix to 1.6 branch to fix the 1.6-1.8 upgrade cases soon.
Sent for 1.6 branch.
Both fixes are merged, I will observe the upgrade suites a bit longer before closing.
Test is passing now (on both 1.7-1.8 and 1.6-1.8, gce and gke). Closing this issue. /close
Awesome!


Question: # We do this in a tmpdir in case the dest has other non-autogenned files # We don't want to include them in the list of gen'd files local tmpdir="${KUBE_ROOT}/doc_tmp" <del> mkdir "${tmpdir}" </del> <ins> mkdir -p "${tmpdir}" </ins> # generate the new files ${cmd} "${tmpdir}" # create the list of generated files

Multiple, for example: Since See above The test tolerations are now to strict: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think it's from if you check the cri-o versions the succeessful one was and the failing was
Working on a fix in
cc
/reopen need to bump in kube as well to fully get the fix
Reopened this issue. <detailsIn response to : /reopen </details


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> # Set provider independent environment variables </del> <ins> # Set the default provider of Kubernetes cluster to know where to load provider-specific scripts # You can override the default provider by exporting the KUBERNETES_PROVIDER # variable in your bashrc # # The valid values: 'gce', 'azure' and 'vagrant' </ins> <del> # Set provider of Kubernetes cluster to know where to load provider-specific scripts, values: gce, vagrant, etc. KUBERNETES_PROVIDER="gce" #KUBERNETES_PROVIDER="vagrant" </del> <ins> KUBERNETES_PROVIDER=${KUBERNETES_PROVIDER:-gce} </ins>

Creating a service with fails to be created when trying to use a TCP health probe. Example service: Failed as follows: Health probe works correctly and is registered properly as TCP health probe. Take the mentioned service yaml and apply it, together with a corresponding pod / deployment exposing port 445. E.g. like this: The respective code seems to take the wrong path to me (it's +/- the same code on the in-tree as on the out of tree cloud provider): My reading of the code and passed values are like this: returns returns -- because returns returns therefore: So the requestPath is not at this point and the azure API server returns a 400 bad request. Kubernetes version (use ): 1.20.6 Cloud provider or hardware configuration: azure OS (e.g: ): linux Network plugin and version (if this is a network-related bug): vxlan & flannel
/sig cloud-provider
related PR introducing the feature:
/assign /triage accepted
/assign


Question: return &http.Response{StatusCode: 200, Header: cmdtesting.DefaultHeader(), Body: cmdtesting.ObjBody(codec, newNode)}, nil case m.isFor("DELETE", "/namespaces/default/pods/bar"): atomic.AddInt32(&deletions, 1) <ins> if test.failUponEvictionOrDeletion { return nil, errors.New("request failed") } </ins> return &http.Response{StatusCode: 204, Header: cmdtesting.DefaultHeader(), Body: cmdtesting.ObjBody(codec, &test.pods[0])}, nil case m.isFor("POST", "/namespaces/default/pods/bar/eviction"): atomic.AddInt32(&evictions, 1) <ins> if test.failUponEvictionOrDeletion { return nil, errors.New("request failed") } </ins> return &http.Response{StatusCode: 201, Header: cmdtesting.DefaultHeader(), Body: cmdtesting.ObjBody(codec, &policyv1beta1.Eviction{})}, nil default: t.Fatalf("%s: unexpected request: %v %#vn%#v", test.description, req.Method, req.URL, req)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl drain crashed with panic when apiserver connection disappeared while there were pods that couldn't be evicted/deleted (sorry, the problem is a bit hard to reproduce and the log was lost) What you expected to happen: kubectl drain should exit with a proper error message How to reproduce it (as minimally and precisely as possible): Try kubectl drain multiple times, interrupting the connection during eviction/deletion Anything else we need to know?: The here can be in case if returns an error, and will panic upon accessing : This causes the c Environment: Kubernetes version (use ): The problem is present as of but is also present in earlier versions (I have a fix for this bug)
/priority important-longterm /sig cli


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package architecture import "github.com/onsi/ginkgo" // SIGDescribe annotates the test with the SIG label. func SIGDescribe(text string, body func()) bool { return ginkgo.Describe("[sig-architecture] "+text, body) } </ins>

I can create a cluster with one node and pass conformance, ~there are some tests that require at least 2 nodes to run~ ~~ ~This is because the tests Skip if they are not able to run~ ~~ ~As consequence, the list of tests for conformance depends on the number of nodes, a single cluster node (that is not a cluster by definition of cluster :)) will run less tests than a multi cluster node and still be Conformant~ EDIT I have to correct myself, it seems that the conformance tests doesn't skip, the problem is: there are no conformance tests that exercise anything that requires more than one node, hence, single node cluster can pass conformance. I expect a cluster (a group of similar things or people positioned or occurring closely together) and a "distributed system" to not be able to be single-node. In addition, lots of most important Kubernetes features can not be exercised on one node, lowering the quality bar. local- pass conformance Related discussion <details</details<details</details<details</details<details</details<details</details<details</details
/assign Maybe this has changed, but last time I've checked the minimum for conformance was 1 control-plane and 2 workers, if that didn't change I will submit a PR to enforce it in the suite Regarding sig-network, with one node you can not verify the networking model, where pod to pod in different nodes should be able to communicate without NAT, that is a strong requirement ... I have to review closely what tests do we have in Conformance
/area conformance /sig network
that seems pretty intentional... changing the definition of conformance to require multiple nodes probably deserves more feedback than an issue/PR
I'm not changing the definition, just asserting it
if we are testing features like replicaset, daemonset, services, ..., in one node only we are missing a lot of testing on those features ... a podman/docker single node can be a kubernetes conformance cluster ... do I need a scheduler or network for anything, I'm just deploying always in the same node?
No I don't think so. Where are your results and configuration? You need two schedulable nodes to pass the conformance suite. That's not correct, we test rolling pod behavior with replicasets. This test requires multiple nodes. There is at least one.

That job is not passing conformance, from the configuration: (see: ) Conformance requires not setting --, you must run all tests without skipping. We don't, at the very least replicaset is being tested multi-node, just not in this job that is not running the full conformance suite. When conformance results are submitted they are now validated to ensure that all tests were run.


Question: }, metricsName: "scheduler_pending_pods", wants: ` <del> # HELP scheduler_pending_pods [ALPHA] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </del> <ins> # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </ins> # TYPE scheduler_pending_pods gauge scheduler_pending_pods{queue="active"} 50 scheduler_pending_pods{queue="backoff"} 0

I would like to propose the graduation of the following scheduler metrics from alpha to stable: These metrics have been stable for at least 15 months.
/sig scheduling /sig instrumentation
/cc
IIRC, the last addition to these metrics was the profile name label. I don't foresee need for new labels. I don't see many precedents for graduation in core components. are there any defined criteria for graduation?
As requested, the following is how long it has been since a change has been made to each of the requested metrics: - 2 years; last change was to migrate to the metrics stability framework - 17 months; last change was a renaming of the metric - 15 months; last change was a renaming of the metric - 17 months; last change was to add a label - 17 months; last change was to add a label
It's up to the component owners to commit to graduating these metrics, since the component owners own the metrics and therefore need to agree to support them in accordance to stability requirements. If you guys are okay with promoting these metrics, I don't see a reason, from an instrumentation side, why we would object to promotion.
+1 to graduation then. These metrics are very mature. Although, I'm not sure if all metrics already adhere to naming conventions. and don't have a unit suffix, but they are unit-less. Is that fine
is a gauge so it's fine, is a histogram so it's also fine.
I'm recently working with metrics, I'm glad to provide some help if no one has interest about it.
There is already an open PR for this.
Oh sorry, didn't notice about that. Forget it.
These graduations sound good to me, but I would be inclined to rename . I feel that the part of the metric is unnecessary. Having instead and keeping for the algorithm latency is already descriptive enough to know the difference between both. If that's not the case, then the text of the metric can be used to guide the users.
I agree that is rather confusing, so +1 to removing the prefix. The metric doesn't include retries, for example. We have a separate metric that includes retries: We need to rename that one eventually to e2e, but it can only be done after the existing one has been deprecated and deleted for a few releases. /triage accepted
/assign
/cc
The PR for this is ready to be reviewed:


Question: // mockgen _ "github.com/golang/mock/mockgen" <ins> // tools like cpu _ "go.uber.org/automaxprocs" </ins> )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : should detect the appropriate GOMAXPROCS based off of the cpu limit supplied to the pod. Why is this needed: If not, test failures can occur (e.g. ) where GOMAXPROCS defaults to the number of cpus on the machine, while the cpu limit is actually far less. Too many compile/test processes launch, and any timing tests can become flaky. The end result is that running the test jobs on a node with more CPUs can make it flaky when running the test on a node with fewer CPUs cleanly passes. It's possible to set GOMAXPROCS=cpu limits in test jobs to avoid this, but it would be nicer if would autodiscover and set GOMAXPROCS appropriately.
/sig testing /sig k8s-infra
/transfer kubernetes
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign dims
xref:
This is already done, which is great, but for some more info I did some benchmarking of setting GOMAXPROCs vs not: Pretty substantially better in some cases


Question: package filters import ( <ins> "context" </ins> "fmt" "net/http" "runtime"

As visible in this graph: ! pod-startup time in 5k-node tests increased around beginning of June (only 99% though). It seems to be connected to "run-to-watch" phase. We should investigate and see what we can do with this. /sig scalability /kind bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : As visible in this graph: ! /kind bugs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Once PR that merged around that time is P&F support for watch-initialitation: As a first step, we may want to confirm/reject if it's this PR and see if we aren't somehow misconfigured here.
I've briefly checked test-infra/perf-tests to see if we had any relevant changes around that time, but I don't see any. Most likely it's related to k/k changes.
/assign
Good news is that we can also see it regression in kubemark (although not as bad as in regular 5k tests): http://perf- But this can help us with finding root cause faster/cheaper. ! I've picked two runs, one before this change, one after: 26th May: Timeline: 12th June: Timeline: First run vs second run I can see significantly higher spike on waiting time in P&F, but according to timeline it seems like it was during huge service and not pod startup latency: 26th May: ! 12th June: ! I can see that number of apiserver goroutines significantly increased (from ~500k to ~900k). I would expect to have 1 more goroutine per watch, we have around 200k watches). ! vs ! Worst 100 pods with slowest latency runtowatch: I've picked the slowest pod "test-2xppkn-33/latency-deployment-60--p7nwz" to analyse.

This pod was bound to node gce-scale-cluster-minion-group-4-txww: Kubelet log for this pod: CL2 logs for this pod: In apiserver logs we can see: So it looks like updating pod status was delayed by P&F by ~500 ms for this pod. Timeline for this pod: 18:37:46. scheduled 18:37:48. container started 18:37:49. kubelet gets pod (1.3s latency with 610 ms delay from P&F) 18:37:50. patch status of pod (545ms latency with 500ms delay from P&F) 18:37:51. observed by CL2 as running To compare with previous run we have this timeline for pod "test-rcdjgd-26/latency-deployment-69--6s95x 2.724031741s": 18:57:08. scheduled on node gce-scale-cluster-minion-group-3-l21s 18:57:10. container started 18:57:10. kubelet gets pod (364 ms latency with 345 ms delay from P&F) 18:57:11. patched status od pod (360 ms latency with 254 ms delay from P&F) So to compare, for these two pods major difference was "get pod" + "patch status" in kubelet: 1.8s vs 700ms. 1s of 1.8s of latency was due to P&F and 0.6s of 0.7s was due to P&F.
In both runs I don't see significant waiting time (99th percentile) on any of Priority levels during pod startup latency phase: 26th May: ! 12th June: !
1) Do we have data about 100% waiting time in P&F? With 99% it my not be enough to to see that data. 2) If really waiting time is small, then the question is what exactly. (a) it might be related to the number of goroutines (b) it might be locking (c) it might be something else... I have just opened the WIP PR: that should help with first two (at least to some extent). Will you be able to test if that helps?

Max waiting time in P&F for pod startup latency phase: 26th May: ! 12th June: ! Buckets are ..., 0.2, 0.5, ... so I don't think it's significant. will test it.
This is definitely non-negligible regression that we should address in 1.22 timeframe. I'm actually have a promissing results for it. /milestone v1.22 /priority critical-urgent
OK - so we proved that addresses the problem: it reduces back to previous value the number of goroutines ! it brings back pod startup time back to less than 3.4s on 99th percentile it brings back API call latencies I'm going to work with reviewers on merging this to fix the regression. - FYI


Question: type HTTPIngressPath struct { // Path is matched against the path of an incoming request. Currently it can // contain characters disallowed from the conventional "path" part of a URL <del> // as defined by RFC 3986. Paths must begin with a '/'. When unspecified, // all paths from incoming requests are matched. </del> <ins> // as defined by RFC 3986. Paths must begin with a '/' and must be present // when using PathType with value "Exact" or "Prefix". </ins> // +optional Path string

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When I tried to use the default path I got . No error. According to the path is not needed. "When unspecified, all paths from incoming requests are matched." Create a ingress like so: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: brew Network plugin and version (if this is a network-related bug): N/A Others:
/sig network /area ingress It seems to be a problem of ingress and I suggest creating an issue in I find a similar one which is a little different there
We use - would this be a problem with the aws repo then?
/assign
So digging a bit here, the API docs needs to be improved. By the , is required if the PathPrefix is Exact or Prefix. To solve your issue specifically, you have two ways: Add that accepts ingress without paths Add and change PathPrefix to Prefix, so this becomes a catchAll I'll open a PR to fix the comment/docs. Thanks!
Thanks! In my case because I'm using ALB controller I had to use with prefix to get it working - see
/triage accepted PR will close this ;)


Question: // updates labels of nodes given by nodeNames. // In case a given label already exists, it overwrites it. If label to remove doesn't exist // it silently ignores it. <ins> // TODO: migrate to use framework.AddOrUpdateLabelOnNode/framework.RemoveLabelOffNode </ins> func updateNodeLabels(c *client.Client, nodeNames sets.String, toAdd, toRemove map[string]string) { const maxRetries = 5 for nodeName := range nodeNames {

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that NodeSelector is respected if matching [Conformance] {Kubernetes e2e suite}


Question: pvc := createPVC("pvc-2", strconv.Itoa(maxPVs/2)+"G", []api.PersistentVolumeAccessMode{api.ReadWriteOnce}) <del> w, _ := testClient.PersistentVolumes().Watch(api.ListOptions{}) </del> <ins> w, err := testClient.PersistentVolumes().Watch(api.ListOptions{}) if err != nil { t.Errorf("Failed to watch PersistentVolumes: %v", err) } </ins> defer w.Stop() for i := 0; i < maxPVs; i++ { <del> _, _ = testClient.PersistentVolumes().Create(pvs[i]) </del> <ins> _, err = testClient.PersistentVolumes().Create(pvs[i]) if err != nil { t.Errorf("Failed to create PersistentVolume %d: %v", i, err) } </ins> } <del> _, _ = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) </del> <ins> _, err = testClient.PersistentVolumeClaims(api.NamespaceDefault).Create(pvc) if err != nil { t.Errorf("Failed to create PersistentVolumeClaim: %v", err) } </ins> // wait until the controller pairs the volume and claim waitForPersistentVolumePhase(w, api.VolumeBound)

PersistentVolume integration tests fail sometimes: There are some interesting reports in the log like: Thans for report.
It turns out that volume integration tests have no logs at all, was not checked and I don't really know what's going on there. adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.
This flake affects also TestPersistentVolumeMultiPVs:
Extended logging got merged just now, please post links to logs when it breaks again!
Marking as P1 as per
I have no real usable logs, but I think it could be fixed by , it hasn't flaked since it was merged. And I have (in merge queue) to further stabilize the tests.
Not blocking this on 1.3.
It hasn't flaked for almost a month, can we close this?
Closing this issue since previous fixes made the flake go away.


Question: // // If the webhook is running within the cluster, then you should use `service`. // <del> // If there is only one port open for the service, that port will be // used. If there are multiple ports open, port 443 will be used if it // is open, otherwise it is an error. </del> <ins> // Port 443 will be used if it is open, otherwise it is an error. </ins> // // +optional Service *ServiceReference

What happened: Deployed a webhook backend and service as follows: Please note that there is only 8443 port on the service. Register the webhook as follows: Got the following error when creating pods: What you expected to happen: Based on , should be called. Not the 443 port shown in the error message. How to reproduce it (as minimally and precisely as possible): Just deploy the same setup and register webhook. Anything else we need to know?: Environment: Kubernetes version (use ): HEAD Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0-alpha.1.97+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2018-01-22T20:00:24Z", GoVersion:"go1.9.1", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-alpha.0.470+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-03-12T19:36:42Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): COS Kernel (e.g. ): COS
thanks for the detailed bug report!
I actually think that the API specification (or documentation) must change. Not all API servers will have access to look up the Service object and see what ports it defines. If we want to support ports other than 443, the port information must be specified in the webhook object, not discovered by direct Service object lookup
/assign
Looks like it will always use 443 no matter what ports are available, right? Should our documentation reflect this or should it behave differently? (also just sets 443 in the custom dial wrapper thing )
As a first step, I would adjust the documentation. As noted in , if we want to support ports other than 443, that information needs to be to the webhook config object


Question: } <del> // FromUnstructuredWIthValidation converts an object from map[string]interface{} representation into a concrete type. </del> <ins> // FromUnstructuredWithValidation converts an object from map[string]interface{} representation into a concrete type. </ins> // It uses encoding/json/Unmarshaler if object implements it or reflection if not. // It takes a validationDirective that indicates how to behave when it encounters unknown fields. func (c *unstructuredConverter) FromUnstructuredWithValidation(u map[string]interface{}, obj interface{}, returnUnknownFields bool) error {

I have created the following secret in my cluster. When I add labels to my secret and try to apply it in the cluster I get the following error This also happens when I use or using the go client. Not sure why, but works. I expect that the secret can be updated with the new labels since the is not actually changed. Steps explained above. No response <details</details<detailsKind </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
/assign
/remove-sig cli /sig apps


Question: volumes                []string } <del> func podWithConfigMaps(ns, name string, toAttach configMapsToAttach) *v1.Pod { </del> <ins> func podWithConfigMaps(ns, podName string, toAttach configMapsToAttach) *v1.Pod { </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Namespace: ns, <del> Name:      name, </del> <ins> Name:      podName, </ins> }, Spec: v1.PodSpec{}, }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Occasionally (more frequently if the node is heavily loaded) the node fails to mount a secret or configmap for a pod. Error recovery fails and the pod must be replaced with a new pod to have any hope of getting the secret or configmap volume mounted. The problem appears to be that if a pod is updated, its secrets are removed from the secretStore's secrets map. See: Normally the effects of the bug are not felt since the usual order of operations masks the bug: 1) The pod is created and registered the first time with the secrets manager 2) The volume mounter gets the secret 3) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) If the race is lost, though, then the bug symptoms are manifest: 1) The pod is created and registered the first time with the secrets manager 2) The pod is updated and registered again with the secrets manager, which deletes the secrets (the bug) 3) The volume mounter tries to get the secret and fails with the message What you expected to happen: On update, the secrets manager should only delete secrets if they are not present in the latest pod definition (rather than deleting all prior secrets). Whether the pod update happens before or after the volume mounter attempts to get the secret, it should succeed. How to reproduce it (as minimally and precisely as possible): Because it is a race condition, it's difficult to reproduce. We've most often seen the bug on systems that were under load. 1) Create a pod that has a secrets or configmap volume 2) See if the pod fails to mount the volume. Anything else we need to know?: If the above analysis is correct, this bug has been present since 1.6 for secret volumes, and 1.7 for configmap volumes. Environment: We've seen the bug in software based upon the 1.6 release.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
The PR addresses this issue. /sig node
This may be a false alarm. I've never actually reproduced it in Kubernetes proper, only in OpenShift. Since our smoking gun was anything but, I'm going to close this until I manage to find an actual problem in Kubernetes.
To close the loop on this for any that find it via future searches: It turns out that this was a false alarm. The above analysis was flawed and this turns out not to be a bug (or maybe a very minor one if we consider it a bug to log a an error on a (semantic) non-error). The error messages like were not an indicator of a failure to mount a secret volume, but instead a failure to update a secret volume during pod teardown. Here's how the error message is triggered: pod is created volume manager gets the secret and mounts the volume the lifetime of the pod, the volume manager periodically re-checks the secret to make sure that it's still in a good state. pod goes away volume is unmounted If one of the refreshes from step 3 happens after step 4 but before step 5, the log message will appear.


Question: // answers dns for service - creates the maximum number of services, and then check dns record for one It("Should answer DNS query for maximum number of services per cluster", func() { <del> services := generateServicesInNamespace(f.Namespace.Name, maxServicesPerCluster) </del> <ins> // get integer ceiling of maxServicesPerCluster / maxServicesPerNamespace numNs := (maxServicesPerCluster + maxServicesPerNamespace - 1) / maxServicesPerNamespace var namespaces []string for i := 0; i < numNs; i++ { ns, _ := f.CreateNamespace(f.BaseName, nil) namespaces = append(namespaces, ns.Name) } services := generateServicesInNamespaces(namespaces, maxServicesPerCluster) </ins> createService := func(i int) { defer GinkgoRecover() <del> framework.ExpectNoError(testutils.CreateServiceWithRetries(f.ClientSet, f.Namespace.Name, services[i])) </del> <ins> framework.ExpectNoError(testutils.CreateServiceWithRetries(f.ClientSet, services[i].Namespace, services[i])) </ins> } framework.Logf("Creating %v test services", maxServicesPerCluster) workqueue.Parallelize(parallelCreateServiceWorkers, len(services), createService)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: For these 2 test suites visible in the testgrid at network-gce : http://k8s- http://k8s- the testsuite cannot setup correctly the cluster, due to a firewall issue (I think) See part of the build log below: What you expected to happen: I would think that setup the cluster is part of the process of the test-infra job, not the e2e test itself. Any missing flag or option on the job definition (see below) ? Need guidance on how to find the root cause of this error. How to reproduce it (as minimally and precisely as possible): Just look at the testgrid, it happen every 6 hours. http://k8s- http://k8s- Anything else we need to know?: job definition is: NOTE: this job definition was copied from another one that seems working fine: Environment: Kubernetes version (use ): current (branch master) Cloud provider or hardware configuration: gce
: I finally create this issue to get help. , : any idea on why the firewall rules cannot be set ? Could you indicate how to investigate this issue ? Thanks..
/sig network /sig scalability
I think it's because the name is too long. There's a limit of 64 chars or sth for resource names on GCP. Try changing the --cluster arg to or sth.
proposed a fix here : it is now merge. Need to verify the next run of these testsuites within 0 to 6 hours from now. : Thanks for your advise !
After updating the name of the cluster (shorter). The Cluster is now setup correctly. The PerformanceDNS test itself is failing - unable to create he 10k services. We believe that is an issue on the number of 10k services that k8s cannot support with this configuration of cluster. Next step is to decrease the number of services to 5k - a number we tested locally and know that k8s support.
Can you tell the reason why it's unable to create so many services?
I suspect that the cluster cannot operate correctly with that many services.

It can create the services. But the cluster acts undesirably after they are created. In my local tests on a single node cluster when I ramped up the number of services to 10K, I saw the same mode of failure. With 10K services, the cluster is unable to spin up the "util" pod. The "util" pod is required in the test to query the DNS from the context of a pod - if it's not running then the test cannot proceed.
But do you know what exactly is happening? What do you mean by "cluster is unable to spin up the "util" pod"? Is the pod not even being created? Or maybe is it scheduled, but kubelet is not starting it? Having more insight would help debugging it (maybe it's just size of master machine that is too small).
I dont for sure. But... According to the logs, I think the pod is successfully created. But when we try to run DNS queries, it's the container in the pod named "util" that cannot be found - not necessarily the pod itself. So, it seems the pod may be still starting up at the time - perhaps slowed by the number of services present. Maybe if I create the pod before creating the services it will help.
Let's try to understand this better. If it's that creating services is limiting us on the api qps, then starting the pod may potentially be slowed down (due to shortage of qps). Your idea sounds good, or you might alternatively try to slow down the rate of service creation. In general, I believe it shouldn't be a problem to create so many empty services in the cluster.
Please paste link to logs from that cluster (run from testgrid) - so that others can take a look into it.
OK, these are logs of the failure when pod is created after services... creates services OK I believe (this step takes just under 10 minutes). Then it creates the client pod... Looks OK? - client pod is created... So we start querying DNS... But the "util" container is not found, possibly because the pod is not ready yet. Perhaps the helper func that I am using doesn't wait for the pod to be ready.
I looked into kubelet logs - pods was succesfully started but then dies (crashed?) - something seems to be wrong on your side.

Logs from here:
It waits for it to be ready, but it dies after that.
OK, thanks but this isnt a pod I have invented for this test. It is already used for testing DNS in other existing tests. So I was expecting it to work. And it does work fine if i reduce the number of services - which I'll go ahead and do.
Let's not down the number of service without understanding why it fails. We need to understand that.
Does anyone know if the 10K limit on services is a hard/enforced limit? I didnt think it was (but thats just an assumption) - but if it is enforced somewhere, and exceeding it causes silent failures, then i'll need to reduce the number of services by a "few" to account for the other already existing services (e.g. the cluster dns service, api, etc...)
It's not a hard limit.
I managed to get this tested locally with 10K services on a single node.... where I create the pod before the services, and it passes. OK if I submit a PR with that change? Or do we want to first try to understand why creating the client pod after services causes a failure?
If you can explain why this works now and didn't work before, I'm fine - otherwise there is a risk that it may e.g. be flaky. We should understand it before blindly changing that.
OK - so I've run it and catched that pod for you. The reason why it is crashlooping is the following log: It just can't initialize itself due to too long argument list. - So I think that the fix should be different. Apparently we are not able to have that many services in a single namespace, because env vars that we provide for all pods (with services from that namespace) are too long in total. So I think what we should do is to: uncover how many services we are able to have in a single namespace (so binary search on number of those should do the job) split those 10k service over multiple namespaces Does that make sense to you? - it's the limitation that recently poped up in our conversations ^^
Thanks Wojtek!

And that is why I said we should make use of the experiments dns folks are already doing, to catch that limitation (and we indeed seem to have caught it) :) On Thu, Jun 7, 2018, 10:47 AM Wojciech Tyczynski <:
I'll update that discussion once the value is figured out. Btw - thanks for digging. On Thu, Jun 7, 2018, 11:03 AM Shyam Jeedigunta <: The reason why it is crashlooping is the following log: exec user process caused "argument list too long" It just can't initialize itself due to too long argument list. So I think that the fix should be different. Apparently we are not able to have that many services in a single namespace, because env vars that we provide for all pods (with services from that namespace) are too long in total. So I think what we should do is to: uncover how many services we are able to have in a single namespace (so binary search on number of those should do the job) split those 10k service over multiple namespaces Does that make sense to you? it's the limitation that recently poped up in our conversations ^^ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
FYI - I just run test with 5000 services and it seemed to work fine (I did just a simple thing to reduce number of services, not split them across namespaces). So I think having 2 namespaces should be fine.
But I got it to work with 10k services in one namespace... by creating the client pod before creating the services. On Jun 7, 2018, at 5:47 AM, Wojciech Tyczynski <mailto:: FYI - I just run test with 5000 services and it seemed to work fine (I did just a simple thing to reduce number of services, not split them across namespaces). So I think having 2 namespaces should be fine. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread
- IIUC that's because you created the pod even before creating the services (so it wasn't passed those service env vars while creation).
OK cool.

Seems like the limit would need to be here ... Several of the entries are "per ns" but there isnt one for services. As an unrelated side note: I just noticed that the limit on the number of deployments per ns in the thresholds doc lists 20K, with a long term goal of 10K. Is that a typo? Or is the long term goal to reduce the limit?
Sent for it. Seems so to me (should've probably been 10^5?). cc
Yeah - I think two values are incorrect wrt it: pods/namespace - I think we've never run 15k deployments/namespace - it doesn't make much sense to support more deployments than pods Will send out a PR for it.


Question: <ins> package(default_visibility = ["//visibility:public"]) </ins> licenses(["notice"]) load(

StatefulSet now as of 1.6. That means the TODO on can be fixed in 1.7. We could not fix it in 1.6 because that code is used by kubectl, which must work against servers that are one minor version ahead of or behind the client.
/cc
I am volunteered to take this job if no one start working on this.
I don't think anyone's working on it, so you are welcome to take it. Feel free to tag me as a reviewer. Thanks!


Question: ready := 0 for i := range pods.Items { pod := pods.Items[i] <del> if podutil.IsPodReady(&pod) { </del> <ins> if podutil.IsPodReady(&pod) && pod.ObjectMeta.DeletionTimestamp.IsZero() { </ins> ready++ } }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: all of the e2e jobs that run this test Which test(s) are flaking: ` Testgrid link: Reason for failure: mostly: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated /sig apps xref:
approx:
my suspicion: /
I can't find anything else that seems relevant in the time frame, and it is the only change to the test file in that time frame.
merged recently, not sure if it's relevant
Not sure what the procedure it, but we can revert or try some other image instead of Pause, if you think it helps. I don't have a good explanation why the change would make the test flake like this.
I don't have a good explanation, just that it affects the test and is in the time frame that seems to fit the flake spike It may take a bit to pin down what exactly happened here :/ I can't look for a little bit.
I tried to reproduce this behaviour on a test cluster in AWS, but couldn't. I prepared a PR to revert the change, just incase.
Good news and bad news kind of thing... The same test fails in the revert PR also, with the previous EchoServer image:
I did an experiment reverting the second suspect in I couldn't reproduce the issue anymore even though I tried. You can check the results in the PR history.
I believe I have found the issue here. More information and fix incoming
Thanks for the update
I just about have a fix ready here. I was able to narrow the commit range to by looking at when this flake first occurred in It looks like was correct that this was introduced in . I believe what is happening is that in the test we are getting getting a with a set (the one that we previously successfully issued an eviction for) before waiting for 3 pods to be ready again in the namespace (). Before the aforementioned PR caused us to skip checking PDBs when is set we would still report back that eviction was not possible because our with did not meet the other criteria (i.e.

was not , , or ), which caused us to still check PDBs, which would show us that we did not have the budget to delete (even though it was not really taking into account the that we were actually attempting to evict). I think there are a few things we can do to address this: we should check that not only is the , but also that it does not have set. above where we call so that we do not attempt to get a before all of the pods are ready. in that the does not have a . Just doing (3) by itself would address our current problem, but (1) and (2) also seem to make sense in the context of what we are trying to check here.
Thanks everyone and apologies for the false alarm :-)
/lifecycle active /priority critical-urgent (this is blocking merges of other fixes) /assign
/milestone v1.19