Question: The runtimeclasses resource MUST exist in the /apis/node.k8s.io/v1 discovery document. The runtimeclasses resource must support create, get, list, watch, update, patch, delete, and deletecollection. */ <del> ginkgo.It(" [NodeConformance] should support RuntimeClasses API operations", func() { </del> <ins> framework.ConformanceIt(" should support RuntimeClasses API operations", func() { </ins> // Setup rcVersion := "v1" rcClient := f.ClientSet.NodeV1().RuntimeClasses()

Per the comment in this PR conformance tests will regress because of a newly Runtime Class GA v1 API is not covered with tests. Full comment: Full comment: /assign More links:
/sig node /triage accepted
KEP:
New to conformance tests. So the idea is to mark tests (or create new ones) that would call each of these methods at lease once? So if this test will be marked as conformance, it will cover and . Is there any additional requirements for the conformance tests? Maybe with the number of side effects of that test or a special execution environment?
Yes. You can make a single test that runs objects through CRUD operations. For the deletecollection one, I'd suggest labeling the objects under test and scoping the deletecollection to your test label to avoid breaking the cluster you're running on. See as an example
Great, will be working on it! Also just chatted with about it.
Just to get clarity on this - is this considered a blocker for Confirming bc grad criteria for it says "No release blocking feedback for API and functionality" should these tests be required to consider this stable? As a note: Code Freeze means that required tests must also be finished:
Here's possibly an easier link to track: !
This seems like something that should be tested generically, or at the very least have a table-driven test that covers all (or most) GA APIs.
! Just a quick note to show that currently the new tests are hitting all endpoints. I'd go ahead and create the Promotion PR in order to get any necessary feedback from #sig-arch
/reopen re: I disagree with the idea that a single test covering CRUD operations means RuntimeClass is now conformance tested. Generic CRUD on a resource's API is not sufficient to describe the intended behavior of the resource. That kind of test is at best verifying that api machinery is working. A vendor would able to pass this test, without implementing any of the behaviors runtimeclass enables. What does do? What kinds of behaviors are visible to an end-user? Are any of those tests in Conformance?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I'm not familiar enough with runtimeclass to know whether these would be a good start:
Thank you for reopening. I agree with this in principle. But I really would appreciate some guidance here. Scheduling tests that takes runtime handler into account are great. However they don't hit any endpoints. I asked and I'm not clear whether those needs to be covered as they are not touching an API. As for actual scheduling, one limitation is posted . Which makes some other tests impossible to include to conformance. I next week, but there may not be all the right people there as it's a Thanksgiving short week. Do you have a suggestion?
I agree that exercising specific CRI implementations is not the place of Conformance testing. Conformance testing is intended to exercise end-user visible behaviors of a Kubernetes cluster, which basically means whatever you can see and do at the Control Plane API level (ref: ) The challenge we face with Conformance testing is that we don't have an easy way to enumerate and measure coverage of behaviors. We experimented with listing behaviors in .yaml so that humans and machines could populate and approve them independent of test implementation, but ultimately abandoned that effort due to its complexity (ref: ) So we use API coverage as an easily machine-measurable/enforceable proxy. Noticing the coverage was at 0% should have been a sign that we should look at the existing test coverage of RuntimeClass and decide which tests were eligible for promotion to Conformance, and whether any new tests needed to be written. In theory a solid test plan for a KEP would include a list of behaviors that need to be tested. That's not immediately clear to me from the KEP for RuntimeClass (ref: ) Based on my read of the KEP, some valid tests might be: tests that exercise the runtimeclass admission controller creating pods with a non-existent runtimeclass creating pods with an existing runtimeclass creating pods with scheduling rules that would clash with runtimeclass scheduling rules creating pods with scheduling rules that would merge with runtimeclass scheduling rules Tests that would be disallowed by conformance requirements because they would require pre-installation/configuration of the cluster under test: running pods with a runtimeclass (unless there's some way to create a functional "dummy" runtimeclass without touching kubelets) running pods with a runtimeclass with a specific handler such as containerd, crio, etc.

and expecting them to run
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
sorry for the delay /remove-lifecycle stale
/area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: nodeNames := e2edaemonset.SchedulableNodes(ctx, cs, ds) e2eskipper.SkipUnlessAtLeast(len(nodeNames), 2, "load-balancer rolling update test requires at least 2 schedulable nodes for the DaemonSet") <ins> if len(nodeNames) > 25 { e2eskipper.Skipf("load-balancer rolling update test skipped for large environments with more than 25 nodes") } </ins> ginkgo.By(fmt.Sprintf("Creating DaemonSet %q", name)) ds, err := cs.AppsV1().DaemonSets(ns).Create(context.TODO(), ds, metav1.CreateOptions{})

master-informing gce-master-scale-correctness Kubernetes e2e suite: [It] [sig-network] LoadBalancers should not have connectivity disruption during rolling update with externalTrafficPolicy=Cluster 01-26 04:01 PST https://k8s- { failed [FAILED] error waiting for daemon pods to be ready: timed out waiting for the condition In [It] at: @ 01/27/23 13:45:59.532 } No response /sig-scalability cc
/assign /cc I warned this test was going to flake ;)
/sig network
Currently flaky so I updated the title but will keep issue open until further stability
yes, agreed
one failure is a GCE problem the other a deployment problem Failures look environmental, but let's keep it open until we get 5/5 greens on this test, there may be a symptom of a regression somewhere else
As per discussion of this issue in the SIG Network meeting today, marking this one as accepted. /triage accepted
The label(s) cannot be applied. These labels are supported: . Is this label configured under or in ? <detailsIn response to : /label kind/flake Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied. These labels are supported: . Is this label configured under or in ? <detailsIn response to : /label flake Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
/remove-kind failing-test
/assign it fails with we need to cap it for large clusters


Question: }, }, }, <ins> { name:     "resource-override", arg:      "dpl", expected: schema.GroupVersionResource{Resource: "deployments", Group: "foo"}, srvRes: []*metav1.APIResourceList{ { GroupVersion: "foo/v1", APIResources: []metav1.APIResource{ { Name:       "deployments", ShortNames: []string{"dpl"}, }, }, }, { GroupVersion: "extension/v1beta1", APIResources: []metav1.APIResource{ { Name:       "deployments", ShortNames: []string{"deploy"}, }, }, }, }, }, </ins> } ds := &fakeDiscoveryClient{}

Copying the issue from Here is my CRD CRD Data Outputs Expected to see the resource list when kubectl get dpl Versions
/sig api-machinery /area custom-resources /kind bug
Can you check if the title is right for this case?
/cc
If you collide with another short name, you have to qualify with a group. It should follows normal resolution rules for priority
this isn't CRD-specific. isn't a colliding shortname. it expanded dpl -deployments, then resolved deployments against preferred group order, instead of limiting resolution to the group(s) that had the shortname
updated description to narrow the report to the case which is a bug
Did a little research, turn out kubectl will take as it's short name: I can give a shot to solve this problem, unless it's by design... : )
FYI. PR .
/assign


Question: package upgrade import ( <ins> "fmt" </ins> "os" "github.com/coredns/corefile-migration/migration"

Any ideas how to solve this?
What does your look like?
/assign
I see that the plugin is present in the config. From what version of Kubernetes are you upgrading from?
I was on 1.15.3 and now trying upgrading to 1.15.4 first. Edit: I'm now on 1.15.4 but upgrade doesn't work to 1.16.0. I don't know what proxy plugin is... must be something that is comming automatic.
The preflight check is showing due to the presence of the plugin, as it has been deprecated. The plugin is to be used in its place. If the is the same as you've mentioned , you can use the while upgrading. The plugin will be replaced to use automatically.
Worked for me. Looks like all services are back up :-) Thanks!
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Having the same issue here. I installed clean k8s 1.13.4 , didn't install or anything. Just upgraded to 1.14.7, then to 1.15.4 and now see the same issue trying to upgrade to 1.16.0
Found and follow ups. removed from configMap kube-system/coredns no more complaints
Seems, I was a little bit to fast. After upgrading I saw a lot of errors in the coredns POD logs and (external) DNS resolution didn't work at all: Then I : Forwarding all requests to the nameservers in the nodes So the solution is replace in : with: see also:
I am having a similar issue, but this is related to rewrite plugin, which I believe is supported in the newer coredns version as well. I can workaround by skipping (or removing rewrite rule and adding it back again). Let me know if you want me to create a new issue to track this, or if this is already tracked. Kubernetes upgrade from 1.15.3 to 1.16.2 kubeadm upgrade plan coreFile
Yes, the upgrade should happen smoothly by just skipping the preflight check.

I’ll improve the functionality of the preflight check since more than a few users are having trouble with this. /reopen
Reopened this issue. <detailsIn response to : Yes, the upgrade should happen smoothly by just skipping the preflight check. I’ll improve the functionality of the preflight check since more than a few users are having trouble with this. /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area kubeadm /sig cluster-lifecycle
Hi , I have the same issue here due to the plugin of coredns that is reported as unsupported. My coredns config: Result of Kubadm version: If I change my coredns configuration to remove usage of plugin, the upgrade plan works: I don't think plugin is unsupported by Coredns, unless I'm missing something.
Hosts is not currently supported by the . However, the presence of a plugin that is not supported by migration library should not be preventing the kubeadm upgrade from proceeding if you use the flag, per .
+1 Lets treat ignoring the error as the workaround users should do, for this unsupported scenario. /close
Closing this issue. <detailsIn response to : +1 Lets treat ignoring the error as the workaround users should do, for this unsupported scenario. </details
Sounds good. FWIW, The original reasoning for requiring the explicit ignoring of the pre-flight check was to make the user aware of the risk that proceeding with the upgrade could result in an invalid Corefile. So, in that sense, its working as intended, but the error message is perhaps too cryptic. Currently: "There are unsupported plugins in the CoreDNS Corefile" What was intended to be conveyed: "There are plugin configurations in the CoreDNS Corefile that cannot be migrated. Continuing with the upgrade without migrating those plugin configurations could result in an invalid Corefile and therefore DNS interruption."
I think improving the message is fine or even enumerating which plugins.
Why was this closed? As I understand the issue, kubeadm fails to upgrade without additional options because of a configuration that was created by a previous kubeadm?
i don't believe this part is true.

the "hosts" plugin is not part of the kubeadm 1.16 manifests, so it's user :
hosts is, but proxy was in the default:
the proxy plugin is mentioned as deprecated:
I know, but ignoring the preflight warning makes the upgrade go through just fine, including migration the ConfigMap. So what is the point of the warning?
I think the migration logic in kubeadm checks the supportability of the current config in all versions from the starting point to the destination - hence kubeadm was alerting incorrectly about proxy, which would have been valid in the current version, but then deprecated/removed by the destination version. It should instead check only the starting point - ensuring that there are no unsupported plugins present in the start version.
please reopen this issue if you think there are PRs that we should sent.
Yes after looking at the related code, it looks like kubeadm uses a "dry run" call to in the preflight check to see if there are any unsupported plugins. It should probably instead call , for the starting version only. /reopen
Reopened this issue. <detailsIn response to : Yes after looking at the related code, it looks like kubeadm uses a "dry run" call to in the preflight check to see if there are any unsupported plugins. It should probably instead call , for the starting version only. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Something didn't seem right about my explanation above so I took a second look - looks like there are two preflight checks. One dry run call to , which is OK as is. There is another preflight check, that calls . It is called with both the starting version and destination version, which checks if the existing config contains any plugins that are unsupported in any version from start to destination. Instead it should be called with only the starting version.
Totally agree with that. When I got the error, I did not know what to do. I had to go on google and hopefully found this post. I removed "proxy" module myself in configuration file, but the issue persisted. I had to look at the code to understand why I got this error. This is not something I should need to do.

Providing the list of unsuported modules would be a good starting point, because we would know why the preflight check is failing. Also, a more explicit error message would be great as well. If I understand well, the message could be something like this: "There are plugin in the CoreDNS Corefile that are not supported by Kubeadm or are deprecated in the next release. Please, review your CoreDNS configuration against Changelog to ensure all your plugins are supported by the new version of CoreDNS" (English is not my native language, so you may found better formulation).
Thanks, Just to summarize, there are two issues raised here: is a in kubeadm's CoreDNS configuration migration preflight check. When the proxy plugin is in use, the preflight check falsely alerts that the proxy plugin is not supported by the configuration migration when in fact it is - the migration will automatically replace proxy with the forward plugin. The workaround for this issue now is to ignore the false warning - i.e. skip the preflight check. plugins that are actually not supported by the configuration migrator (e.g. hosts for now), the warning is cryptic and should be expanded/rephrased to more clearly convey the following: The corefile contains plugins that kubeadm/CoreDNS does not know how to migrate. A list of those plugins should be provided in the preflight check message. To be safe, each plugin listed should be manually verified for compatibility with the newer version of CoreDNS. Once ready, the upgrade can be initiated by skipping the preflight check. During the upgrade, kubeadm will migrate the configuration while leaving the listed plugin configs untouched, but cannot guarantee that they will work with the newer version of CoreDNS. Perhaps a link to more detailed documentation if it gets too hard to succinctly explain this (could go in coredns/corefile-migration).


Question: } } <del> func TestPodPhaseWithRestartAlways(t *testing.T) { desiredState := api.PodSpec{ Containers: []api.Container{ {Name: "containerA"}, {Name: "containerB"}, }, RestartPolicy: api.RestartPolicy{Always: &api.RestartPolicyAlways{}}, } currentState := api.PodStatus{ Host: "machine", } runningState := api.ContainerStatus{ State: api.ContainerState{ Running: &api.ContainerStateRunning{}, }, } stoppedState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{}, }, } tests := []struct { pod    *api.Pod status api.PodPhase test   string }{ {&api.Pod{Spec: desiredState, Status: currentState}, api.PodPending, "waiting"}, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": runningState, }, Host: "machine", }, }, api.PodRunning, "all running", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": stoppedState, "containerB": stoppedState, }, Host: "machine", }, }, api.PodRunning, "all stopped with restart always", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": stoppedState, }, Host: "machine", }, }, api.PodRunning, "mixed state #1 with restart always", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, }, Host: "machine", }, }, api.PodPending, "mixed state #2 with restart always", }, } for _, test := range tests { if status := getPhase(&test.pod.Spec, test.pod.Status.Info); status != test.status { t.Errorf("In test %s, expected %v, got %v", test.test, test.status, status) } } } func TestPodPhaseWithRestartNever(t *testing.T) { desiredState := api.PodSpec{ Containers: []api.Container{ {Name: "containerA"}, {Name: "containerB"}, }, RestartPolicy: api.RestartPolicy{Never: &api.RestartPolicyNever{}}, } currentState := api.PodStatus{ Host: "machine", } runningState := api.ContainerStatus{ State: api.ContainerState{ Running: &api.ContainerStateRunning{}, }, } succeededState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ ExitCode: 0, }, }, } failedState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ ExitCode: -1, }, }, } tests := []struct { pod    *api.Pod status api.PodPhase test   string }{ {&api.Pod{Spec: desiredState, Status: currentState}, api.PodPending, "waiting"}, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": runningState, }, Host: "machine", }, }, api.PodRunning, "all running with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": succeededState, "containerB": succeededState, }, Host: "machine", }, }, api.PodSucceeded, "all succeeded with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": failedState, "containerB": failedState, }, Host: "machine", }, }, api.PodFailed, "all failed with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": succeededState, }, Host: "machine", }, }, api.PodRunning, "mixed state #1 with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, }, Host: "machine", }, }, api.PodPending, "mixed state #2 with restart never", }, } for _, test := range tests { if status := getPhase(&test.pod.Spec, test.pod.Status.Info); status != test.status { t.Errorf("In test %s, expected %v, got %v", test.test, test.status, status) } } } func TestPodPhaseWithRestartOnFailure(t *testing.T) { desiredState := api.PodSpec{ Containers: []api.Container{ {Name: "containerA"}, {Name: "containerB"}, }, RestartPolicy: api.RestartPolicy{OnFailure: &api.RestartPolicyOnFailure{}}, } currentState := api.PodStatus{ Host: "machine", } runningState := api.ContainerStatus{ State: api.ContainerState{ Running: &api.ContainerStateRunning{}, }, } succeededState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ ExitCode: 0, }, }, } failedState := api.ContainerStatus{ State: api.ContainerState{ Termination: &api.ContainerStateTerminated{ ExitCode: -1, }, }, } tests := []struct { pod    *api.Pod status api.PodPhase test   string }{ {&api.Pod{Spec: desiredState, Status: currentState}, api.PodPending, "waiting"}, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": runningState, }, Host: "machine", }, }, api.PodRunning, "all running with restart onfailure", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": succeededState, "containerB": succeededState, }, Host: "machine", }, }, api.PodSucceeded, "all succeeded with restart onfailure", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": failedState, "containerB": failedState, }, Host: "machine", }, }, api.PodRunning, "all failed with restart never", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, "containerB": succeededState, }, Host: "machine", }, }, api.PodRunning, "mixed state #1 with restart onfailure", }, { &api.Pod{ Spec: desiredState, Status: api.PodStatus{ Info: map[string]api.ContainerStatus{ "containerA": runningState, }, Host: "machine", }, }, api.PodPending, "mixed state #2 with restart onfailure", }, } for _, test := range tests { if status := getPhase(&test.pod.Spec, test.pod.Status.Info); status != test.status { t.Errorf("In test %s, expected %v, got %v", test.test, test.status, status) } } } </del> func TestGarbageCollection(t *testing.T) { pod1 := makePod(api.NamespaceDefault, "foo", "machine", "bar") pod2 := makePod(api.NamespaceDefault, "baz", "machine", "qux")

Kubelet now only reports ContainerStatus. PodStatus / PodCondition is observed at master side based on a set of ContainerStatus. There is no other way to get PodIP except net container for master side. This is very confusion to the end user, , , ... Now we are finalizing on v1beta3 API, including PodStatus & ContainerStatus. We can make net container completely invisible to the end users by finishing the following steps: 1) Expose network related errors at API level 2) Generate PodStaus directly from Kubelet
3) once PodStatus generated by kubelet, PodIP can be removed from ContainerStatus.
I think reporting PodStatus rather than just container status is most important. How hard is that?
Used to be hard since there are many dependencies when I first report Status back. Now it is totally feasible. I already started to write PRs for this. :-)
One more PR required to close this one.


Question: // newUnmounterInternal is the internal unmounter routine to clean the volume. func (plugin *flexVolumePlugin) newUnmounterInternal(volName string, podUID types.UID, mounter mount.Interface, runner exec.Interface) (volume.Unmounter, error) { <ins> var metricsProvider volume.MetricsProvider if plugin.capabilities.SupportsMetrics { metricsProvider = volume.NewMetricsStatFS(plugin.host.GetPodVolumeDir( podUID, utilstrings.EscapeQualifiedNameForDisk(plugin.driverName), volName)) } else { metricsProvider = &volume.MetricsNil{} } </ins> return &flexVolumeUnmounter{ flexVolume: &flexVolume{ <del> driverName: plugin.driverName, execPath:   plugin.getExecutable(), mounter:    mounter, plugin:     plugin, podUID:     podUID, volName:    volName, </del> <ins> driverName:      plugin.driverName, execPath:        plugin.getExecutable(), mounter:         mounter, plugin:          plugin, podUID:          podUID, volName:         volName, MetricsProvider: metricsProvider, </ins> }, runner: runner, }, nil

kind feature What happened: Currently there is no mechanism to report metrics as the flex plugins are not enabled to collect metrics. All flex volumes lack this feature as compared to in-tree/built-in drivers even though it is possible to collect such metrics for most of the plugins. What you expected to happen: Depending on the availability of the metrics from the mounted FS, flex plugins should allow to report metrics for the FS. How to reproduce it (as minimally and precisely as possible): The default behavior that can be proved easily by just running a command similar to the following command on a node that is running a flex volume. This will show that there are no metrics collected for flex volumes. kubectl proxy $curl 127.0.0.1:8001/api/v1/nodes/<IP/proxy/stats/summary Environment**: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: local x86 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux KubeStorage 4.4.0-130-generic -Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Others: /sig storage
+1 Yes, it will be good, if flex volume can support atleast the default metrics( i.e ), which most of the volume plugins support now. Based upon flex plugin "capabilities" parameters, it can be decided, whether the plugin supports "default metrics" or "nil metrics".


Question: PodWaiting PodStatus = "Waiting" // PodRunning means that the pod is up and running. PodRunning PodStatus = "Running" <del> // PodTerminated means that the pod has stopped. </del> <ins> // PodTerminated means that the pod has stopped with error(s) </ins> PodTerminated PodStatus = "Terminated" // PodUnknown means that we failed to obtain info about the pod. PodUnknown PodStatus = "Unknown" <ins> // PodSucceeded means that the pod has stopped without error(s) PodSucceeded PodStatus = "Succeeded" </ins> ) type ContainerStateWaiting struct {

This happened to me several times while debugging an e2e test. My branch was based on commit and had changes only to
docker inspect says the container exited 0
kubelet HTTP says the pod exited 0:
kubectl says it failed:
Events does not show any success or failure even. Seems like it doesn't show that for any pods, succeeded or failed.
Apiserver sees the exit code 0:
The problem might be related to there being zero values for startedAt or finishedAt. The command may have exited so quickly that docker could not record these values, and we may be somehow treating unset values for these as failure.
adding sleep 5 to my command causes docker to set the startedAt and finishedAt to reasonable values, but the pod still shows up as failed. Note that I use a different pod name each time I try to avoid that sort of obvious mistake.
So, the apiserver v1beta1 message knows that it is terminated with exit code 0. But, both kubectl and my go code in my go e2e test thing this means failed. I'm starting to suspect the conversion code.
The problem is that a v1beta1 pod can never see that a pod is failed or succeded. So it defaults to PodFailed. Grrrrrrraaaaahhhhhh!
In line 184 of
This is going to be confusing for people unless we change this.
I believe the latest fix for PodStatus should resolve the issue here. I will double confirm it before closing it.

We should do this after 1.7. xref /sig testing cc
Is this the task that needs completed - Add support to Ubuntu e2e test-infra
That is different, this is about running GPU tests as part of PR pre-submit tests. Right now we skip them because they are "[Serial]".
I can work on this issue in conjunction with . Is this issue just a matter of adding GPU tests to a different suite or are there more subtasks?
Anymore progress on this?
Thanks for the reminder. I just submitted an internal request for quota to run these tests. Once that is approved, I will add a PR in test-infra to run GPU tests on all k/k PRs. It will be non-blocking initially and then become blocking later on.
Awesome, thanks!


Question: // in case of latency in our caches, make a call direct to storage to verify that it truly exists or not namespaceObj, err = l.client.Namespaces().Get(a.GetNamespace()) if err != nil { <del> return admission.NewForbidden(a, fmt.Errorf("Namespace %s does not exist", a.GetNamespace())) </del> <ins> return admission.NewNotFound(a) </ins> } }

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: nodes, err := c.CoreV1().Nodes().List(ctx, metav1.ListOptions{}) framework.ExpectNoError(err) nodeCount := len(nodes.Items) <del> retryTimeout := dsRetryTimeout + time.Duration(nodeCount*30)*time.Second </del> <ins> // We disturb daemonset progress by randomly terminating pods. randomPodTerminationTimeout := 5 * time.Minute retryTimeout := dsRetryTimeout + randomPodTerminationTimeout + time.Duration(nodeCount*30)*time.Second </ins> ginkgo.By("Check that daemon pods surge and invariants are preserved during that rollout") <del> ageOfOldPod := make(map[string]time.Time) </del> <ins> nodeToAgeOfOldPod := make(map[string]map[string]time.Time) </ins> deliberatelyDeletedPods := sets.NewString() err = wait.PollUntilContextTimeout(ctx, dsRetryPeriod, retryTimeout, true, func(ctx context.Context) (bool, error) { podList, err := c.CoreV1().Pods(ds.Namespace).List(ctx, metav1.ListOptions{})

/kind flake /sig apps This is an old flaking since , but the flaky rate is not that high. The rate in recent runs in is about 3/23 ~ 13%.
/priority backlog as the failure rate now is not high /cc old threads in and
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority backlog as the failure rate now is not high /cc old threads in and Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority backlog
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten still flaking
/triage accepted /assign


Question: delete(quotaPodMap, podQuotaMap[poduid]) delete(podDirCountMap, poduid) delete(podQuotaMap, poduid) <ins> delete(podUidMap, poduid) </ins> } else { err = removeProjectID(path, projid) podDirCountMap[poduid]--

I have kubernetes 1.24.7. I tried to use on formated volume. It works until first reboot. After reboot, I get messages: I expect that project quota can be assigned even after reboot. Use xfs for /var/lib/containerd with discard,quota,prjquota No response <details</details<detailsbaremetal </details<details</details<details</details<detailsrke2 </details<details</details
/sig node Can this be reproduced? Can this pod be recreated correctly if you force delete the pod?
/triage accepted
It can be reproduced. Pod is created if I remove . I can test force delete.
/assign we are looking into gate to understand if we need to keep it or deprecate it.
If I force delete pod, it does not start again:
/assign Let me check on the latest code with the fix


Question: import ( "fmt" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> ) var (

Part of See example /sig scheduling /kind cleanup /help /priority important-soon
/assign /remove-help


Question: import ( "fmt" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> ) var (

Part of /priority important-soon See example /sig scheduling /kind cleanup
/help
/assign
/remove-help


Question: ginkgo.It("should delete a collection of pods", func() { podTestNames := []string{"test-pod-1", "test-pod-2", "test-pod-3"} <ins> zero := int64(0) </ins> ginkgo.By("Create set of pods") // create a set of pods in test namespace for _, podTestName := range podTestNames {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce (5% of runs) pull-kubernetes-e2e-kind-ipv6 pull-kubernetes-e2e-gce-ubuntu-containerd Which test(s) are flaking: This is the flakiest test in the e2e job according to Testgrid link: Reason for failure: Anything else we need to know:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
merged 2 weeks ago in
/assign
potential mitigation in
To track down the test flaking there was some great feedback in by and was created as a potential fix based on that feedback /cc


Question: // the specified LimitRange.  The pod may be modified to apply default resource // requirements if not specified, and enumerated on the LimitRange func PodLimitFunc(limitRange *api.LimitRange, pod *api.Pod) error { <ins> var errs []error </ins> defaultResources := defaultContainerResourceRequirements(limitRange) mergePodResourceRequirements(pod, &defaultResources) for i := range limitRange.Spec.Limits { limit := limitRange.Spec.Limits[i] limitType := limit.Type // enforce container limits if limitType == api.LimitTypeContainer { for j := range pod.Spec.Containers { container := &pod.Spec.Containers[j] for k, v := range limit.Min { if err := minConstraint(limitType, k, v, container.Resources.Requests, container.Resources.Limits); err != nil { <del> return err </del> <ins> errs = append(errs, err) </ins> } } for k, v := range limit.Max { if err := maxConstraint(limitType, k, v, container.Resources.Requests, container.Resources.Limits); err != nil { <del> return err </del> <ins> errs = append(errs, err) </ins> } } for k, v := range limit.MaxLimitRequestRatio { if err := limitRequestRatioConstraint(limitType, k, v, container.Resources.Requests, container.Resources.Limits); err != nil { <del> return err </del> <ins> errs = append(errs, err) </ins> } } }

When a resource violates the limitrange, both for cpu and memory, only one error will be returned at a time. That means when a user is creating such a resource, first it will fail and return something like , them user may edit the cpu usage and create again, then fail again and get . It seem admission controls want to reject the request as fast as possible, if error occurs. But in this case, it may give users a bad experience. Would it be better to aggregate all errors and return them at one time. Also, the error message may cause misunderstanding. , it means the resource requested is 3, it may also cause a confusion that it is limited to 3.
Closing since it's fixed


Question: "net/http/httptest" "reflect" "testing" <ins> "time" </ins> "github.com/gogo/protobuf/proto" "github.com/googleapis/gnostic/OpenAPIv2"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


Question: ) type fakePodKiller struct { <del> killedPods []*v1.Pod </del> <ins> killedPods          []*v1.Pod errDuringPodKilling bool </ins> } <del> func newFakePodKiller() *fakePodKiller { return &fakePodKiller{killedPods: []*v1.Pod{}} </del> <ins> func newFakePodKiller(errPodKilling bool) *fakePodKiller { return &fakePodKiller{killedPods: []*v1.Pod{}, errDuringPodKilling: errPodKilling} </ins> } func (f *fakePodKiller) clear() {

What happened: Kops runs from a pod manifest stored on the node, having the annotation. We've experienced the case where the cluster autoscaler turned on a new node and, once the node has been to the cluster, application pods have been admitted before . Once it was the turn of , all system resources were utilized, so the pod has not been regularly admitted. However, since it's a critical pod, the executed the , which has tried to kill () a pod whose container wasn't started yet (it was downloading the Docker image), which has caused to timeout and return the error . The admission of failed with the following error: The pod execution has not been retried, so we've got a node without running and its networking broken. What you expected to happen: The should be eventually admitted. Anything else we need to know?: A minimal log extraction for comprehension: From the "Failed to admit pod" log I have been able to trace back the core code involved: () Logs: "Failed to admit pod [POD] - [ERROR]" () Error message: "preemption: pod [POD] failed to evict [ERROR]" () -() Error message: "timeout waiting to kill pod" Environment: Kubernetes version: Cloud provider: AWS OS: ubuntu 16.04.6 LTS (Xenial Xerus) Kernel: Install tools: kops Network plugin and version: kubenet
/sig node
I think the following change may alleviate the situation you encountered:
Thanks for your reply. To my understanding, when the is called it will return the minimum set of pods required to free up resources, so even if only 1 pod eviction fails it's likely that at the end of the the will not have enough resources anyway to admit the critical pod. In your opinion / experience: be an option introducing retries in the in case the eviction of 1 pod fails? yes, do you have any suggestion of a similar design adopted somewhere else in the ?
Looking at getPodsToPreempt, I am not sure it returns the minimum set of Pods. For , retrying may not help much since the pod to be killed may not respond to the subsequent kill command.
Sorry to reiterate on this, but I double checked the code and - if I'm not missing anything - returns the minimum set of pods.

Let's look at logic: pods by QOS class using that the total sum of evictable pods resources is enough to satify the requirements (resources to evict for the critical pod) the list of pods to evict, as the merge of three sublists: : the minimum set of pods to evict assuming we've already evicted all and all : the minimum set of pods to evict assuming we've already evicted all and the required : the minimum set of pods to evict to satify the requirements left Am I missing anything? A possible different approach to solve the issue is to refactor to return all input pods sorted by "should be evicted first". Then iterates over the full sorted list of potential pods to evict until the requirements are not satisfied (skipping to subtract pod resources from requirements if a pod fails on ) and breaking the loop as soon as the requirements are satisfied. What's your take on this? /cc who originally implemented this
w.r.t. approach outlined above, it makes sense.
Whether this evicts the "minimum set" depends on how you define the "minimum set". At the time this was implemented (k8s v1.5.0 IIRC), we didn't have PodPriority, so we used QoS to mirror the kubelet's eviction algorithm. It should probably prioritize low-priority pods for eviction today given the recent changes to the eviction algorithm to incorporate PodPriority. The current approach minimizes the eviction of guaranteed pods, and prioritizes evicting few larger pods over many smaller ones. Since eviction marks the evicted pod as failed, it should suffice to just admit the pod regardless of what is returned by the evictPod function. In future syncPod loops, the kubelet will retry the pod deletion steps that it was stuck on. It would eliminate this class of bugs entirely, which was the purpose of the feature in the first place.
See if the following aligns with your suggestion:
Something like that, yeah. I'm not sure the "Failed to admit pod" log warning is still required, but we can iron those things out during code review.
Doing like this it means we do always admit pods, while we should do it only for critical pods, correct?
Please see this The change is for critical pod(s).


Question: done # Run gazelle to update Go rules in BUILD files. <ins> # filter out known pkg-config error (see buildozer workaround below) # NOTE: the `|| exit "${PIPESTATUS[0]}"` is to ignore grep errors # while preserving the exit code of gazelle </ins> gazelle fix  -external=vendored  -mode=fix  -repo_root "${KUBE_ROOT}"  <del> "${KUBE_ROOT}" </del> <ins> "${KUBE_ROOT}"  2>&1 | grep -Ev "vendor/github.com/seccomp/libseccomp-golang/seccomp(_internal)?.go: pkg-config not supported: #cgo pkg-config: libseccomp" || (exit "${PIPESTATUS[0]}") </ins> # Run kazel to update the pkg-srcs and all-srcs rules as well as handle # Kubernetes code generators.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Got the following error while trying to run : How to reproduce it (as minimally and precisely as possible): Ran this on a fresh install of ubuntu 19.04, Go 1.12.6, and bazel 0.26.1. Anything else we need to know?: Seems like the change might have been due to the update in vendored deps from ( ). Although this script is run in CI where it is working so may be some set of recommende pkg versions would help prevent this problem? Environment: Cloud provider or hardware configuration: OS (e.g: ): ubuntu 19.04 Kernel (e.g. ): 5.0.0-15-generic bazel: 0.26.1 Go: 1.12.6
Not sure who owns this... /sig testing
Seems like the errors are "warnings" and can be ignored? It seems that BUILD files are properly updated by update-bazel, And bazel builds do work properly... /cc
/sig release this is not a failure /retitle hack/update-bazel produces warnings there was some discussion on the original PR about filtering them out but nothing conclusive I'll file a PR to do that now.. /assign
aside: this is actually from :-) we're working around the lack of support because we only need it for this package and on all known systems we compile on so far it only needs the one link flag as previously specified in the cgo config.
filed
Glad to see it. The output message is so misleading that I have spent hours on this issue
sorry about that the output does say "error" but it's actually considered a non-fatal warning in the upstream code and doesn't cause the script (or build) to fail.
was merged before this discussion was resolved , so we kinda saw this coming. :/ Hopefully we can get the fix from in merged as quick as possible
/reopen we have a new a new occurrence with libipmctl
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have the same question. This output is a bit annoying.
We can update the script to handle this issue and suppress the warning. The previous PR is an example. On Thu, Apr 16, 2020, 12:14 AM <:
/assign


Question: ginkgo.It("should have stable networking for Linux and Windows pods", func() { ginkgo.By("creating linux and windows pods") linuxPod := createTestPod(f, linuxBusyBoxImage, linuxOS) <ins> linuxPod = f.PodClient().CreateSync(linuxPod) </ins> windowsPod := createTestPod(f, windowsBusyBoximage, windowsOS) windowsPod.Spec.Containers[0].Args = []string{"test-webserver"} <ins> windowsPod = f.PodClient().CreateSync(windowsPod) </ins> ginkgo.By("checking connectivity to 8.8.8.8 53 (google.com) from Linux") assertConsistentConnectivity(f, linuxPod.ObjectMeta.Name, linuxOS, linuxCheck("8.8.8.8", 53))

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: All the v1.18 / master jobs in: Which test(s) are failing: [sig-windows] Hybrid cluster network for all supported CNIs should have stable networking for Linux and Windows pods Since when has it been failing: Since this was merged: Testgrid link: Reason for failure: Previously, we've centralized several images into agnhost, including test-webserver. The Hybrid cluster network test was using the test-webserver image, and was updated to use agnhost, but without properly making it so it behaves like test-webserver, resulting in a failing test. Anything else we need to know: /sig windows /sig testing /priority important-soon


Question: // Search returns a list of events matching the specified object. func (c *FakeEvents) Search(scheme *runtime.Scheme, objOrRef runtime.Object) (*v1.EventList, error) { <del> action := core.NewRootListAction(eventsResource, eventsKind, metav1.ListOptions{}) </del> <ins> var action core.ListActionImpl </ins> if c.ns != "" { action = core.NewListAction(eventsResource, eventsKind, c.ns, metav1.ListOptions{}) <ins> } else { action = core.NewListAction(eventsResource, eventsKind, v1.NamespaceDefault, metav1.ListOptions{}) </ins> } obj, err := c.Fake.Invokes(action, &v1.EventList{}) if obj == nil {

Opening this issue based on what I ran into on with namespace as follows While unit testing the controller, we create a fake client and setup the controller to run using the client. This ends up creating a having the namespace set to When the call eventually reaches The action gets associated with a namespace only if the object has a namespace associated with it. Which in this will be Which will result in the error in to manage the event like does. In case if the namespace is set to Above is a test case that can easily simulate the issue in question. Referring back to the test under PR NA <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: "claim-reference": { pod:    podWithClaimName, claims: []*resourcev1alpha2.ResourceClaim{allocatedClaim, otherClaim}, <ins> want: want{ reserve: result{ changes: change{ claim: func(claim *resourcev1alpha2.ResourceClaim) *resourcev1alpha2.ResourceClaim { if claim.Name == claimName { claim = claim.DeepCopy() claim.Status.ReservedFor = inUseClaim.Status.ReservedFor } return claim }, }, }, }, </ins> }, "claim-template": { pod:    podWithClaimTemplate, claims: []*resourcev1alpha2.ResourceClaim{allocatedClaim, otherClaim}, <ins> want: want{ reserve: result{ changes: change{ claim: func(claim *resourcev1alpha2.ResourceClaim) *resourcev1alpha2.ResourceClaim { if claim.Name == claimName { claim = claim.DeepCopy() claim.Status.ReservedFor = inUseClaim.Status.ReservedFor } return claim }, }, }, }, </ins> }, "missing-claim": { pod: podWithClaimTemplate,

Apply this patch: Then run: This fails with: Test should pass. See above. If there are N objects in claims.Items, then this loop for _, claim := range claims.Items { objects = append(objects, &claim) } is appending the same pointer N times, pointing at the final item in the list. Adding fixes that mistake, but then it makes the test fail. Something else about the test is buggy, but the bug was being masked by this bug. In the repro case I to the loop a few lines down, just for good measure, but that one is not causing a problem. The line by itself causes the test failure. HEAD N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling /priority important-soon /cc Thanks for finding this and the detailed report
/assign
Can you help to fix this?
If we don't here from soonish, then I'll fix it myself.
Sorry, I've been busy with an office accident this week, if you want fix it, please take it
/assign
/unassign


Question: }, metricsName: "scheduler_pending_pods", wants: ` <del> # HELP scheduler_pending_pods [ALPHA] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </del> <ins> # HELP scheduler_pending_pods [STABLE] Number of pending pods, by the queue type. 'active' means number of pods in activeQ; 'backoff' means number of pods in backoffQ; 'unschedulable' means number of pods in unschedulableQ. </ins> # TYPE scheduler_pending_pods gauge scheduler_pending_pods{queue="active"} 30 scheduler_pending_pods{queue="backoff"} 20

I would like to propose the graduation of the following scheduler metrics from alpha to stable: These metrics have been stable for at least 15 months.
/sig scheduling /sig instrumentation
/cc
IIRC, the last addition to these metrics was the profile name label. I don't foresee need for new labels. I don't see many precedents for graduation in core components. are there any defined criteria for graduation?
As requested, the following is how long it has been since a change has been made to each of the requested metrics: - 2 years; last change was to migrate to the metrics stability framework - 17 months; last change was a renaming of the metric - 15 months; last change was a renaming of the metric - 17 months; last change was to add a label - 17 months; last change was to add a label
It's up to the component owners to commit to graduating these metrics, since the component owners own the metrics and therefore need to agree to support them in accordance to stability requirements. If you guys are okay with promoting these metrics, I don't see a reason, from an instrumentation side, why we would object to promotion.
+1 to graduation then. These metrics are very mature. Although, I'm not sure if all metrics already adhere to naming conventions. and don't have a unit suffix, but they are unit-less. Is that fine
is a gauge so it's fine, is a histogram so it's also fine.
I'm recently working with metrics, I'm glad to provide some help if no one has interest about it.
There is already an open PR for this.
Oh sorry, didn't notice about that. Forget it.
These graduations sound good to me, but I would be inclined to rename . I feel that the part of the metric is unnecessary. Having instead and keeping for the algorithm latency is already descriptive enough to know the difference between both. If that's not the case, then the text of the metric can be used to guide the users.
I agree that is rather confusing, so +1 to removing the prefix. The metric doesn't include retries, for example. We have a separate metric that includes retries: We need to rename that one eventually to e2e, but it can only be done after the existing one has been deprecated and deleted for a few releases. /triage accepted
/assign
/cc
The PR for this is ready to be reviewed:


Question: return nil } <del> // dockerJSONLog is the JSON log buffer used in parseDockerJSONLog. var dockerJSONLog = &jsonlog.JSONLog{} </del> // parseDockerJSONLog parses logs in Docker JSON log format. Docker JSON log format // example: //   {"log":"content 1","stream":"stdout","time":"2016-10-20T18:39:20.57606443Z"} //   {"log":"content 2","stream":"stderr","time":"2016-10-20T18:39:20.57606444Z"} func parseDockerJSONLog(log []byte, msg *logMessage) error { <del> dockerJSONLog.Reset() l := dockerJSONLog </del> <ins> var l = &jsonlog.JSONLog{} l.Reset() </ins> // TODO: JSON decoding is fairly expensive, we should evaluate this. if err := json.Unmarshal(log, l); err != nil { return fmt.Errorf("failed with %v to unmarshal log %q", err, l)

I was running a kubectl logs -f, not for long, that broke unexpectedly with Doesn't seem normal.
I'm seeing this on 1.6.2
do you use docker, if so, what log driver do you use? If you use the default log driver and have access to the node that the specific container ran on, it'd be great if you can provide content of the raw log file (located in ).
I saw this in a GKE cluster so I don't have access to the node....
Whose gke cluster? If you are the owner of the gke cluster, you do have access to the node VMs. You can into them.
So I have access after all :) docker 11.2 is used and the log driver is json-file. Unfortunately, I have scaled the deployment down since I had the issue and now I cannot find the container log under /var/log/containers. Noted for the next time tho
docker 11.2.6 using json-file I'll grab the log off the host next time i see it. I assume just like 20 lines above and below where kubectl logs fails would be sufficient?
Yes, that'd be great.
In case it helps someone else, here's how I grabbed the logs in question as is installed on the host The client-side experience the output from and the from that container's (mildly redacted, and I know you asked for about 20 lines, but the lines above those are some sensitive bits that I would prefer not to have to go through and redact; but I will if you think it's necessary) I have the contents of , too, but they are md5-level the exact same content Environmental information regrettably_ we are using the based from CoreOS, invoked as such:
Given , I really don't understand why this happens. The only possibility is that kubelet is reading and parsing a partial line. However, if the json log line is in-complete, should return error and kubelet will discard that line; Or else, it means that json is complete, and I don't understand why is empty when the actual log line contains everything. Could you check kubelet log to see whether there is anything suspicious, although unlikely.

sorry for the delay in getting back to you, I finally caught something interesting in the kubelet log I elided the info level lines in between those error level lines, as they were "MountVolume.Setup succeeded" and didn't seem relevant I also came here to see if there was another issue regarding intermingled log output, but searching for that in the issues is extremely hard given how many ways there are in english to describe the behavior: where other-pod-line-1 is very clearly the log output from a separate Pod on that same Node
I think this is a concurrency issue. I've not been able to reliably reproduce it but anecdotally have noticed that the logging is much less stable when multiple people are tailing at once. I can tail logs off-hours when no one is around and it will be fine for hours without this error. But during the work day when multiple people are tailing the same log we all will all see the error quite regularly. Pretty much every time the error happens, if I ask around it turns out someone else had just started tailing the same log.
In my case, I doubt that it is multiple people tailing simultaneously, but I will try to remember to start several and see if that impacts things I experience it pretty regularly, but I haven't tried reproducing it in any self-contained system -- there are so many variables to consider :( do you also experience that log lines are intermixed with other Pod's output?
I've never seen that. We have a few hundred pods, and haven't witnessed it once or seen anyone else complain about it. But if somehow the streams are being crossed for you, that could also point to a concurrency issue possibly. In your case it may not be multiple readers, but possibly multiple writers. I've not looked at the code though, so just a guess.
Thanks for your suggestion, Bryan, as I am able to blow up the kubelet at will now; I get one of these explosions for each kubectl that attaches: I'll see if I can get it to explode in a more isolated environment, since it doesn't appear to be related to the pods that are running
I see this bug frequently when calling many times in parallel on my kubeadm Kubernetes 1.7.4 cluster.

I was using multiple threads (from a Python3 process) to get many logs at once using kubectl log, and that would cause me to hit this bug on about 25% of the requests. Switching to grabbing the logs one after the other (rather than doing many kubectl log shell commands at once) works around the problem, though is much an order of magnitude slower. To be clear: I am not using the follow option. I am using the tail option, typically .
Also reproduced the problem by fetching container logs in multiple processes. should fix the problem.
I am able to reproduce this with minikube v0.22.1 (which runs kubernetes 1.7.5). When following logs for separate pods concurrently (using ), I very quickly see . Funny thing is there's always one random pod for which the follow doesn't crash with this error (at least not for a long time) and keeps running successfully.
Seeing this error message with a 1.7.3 bare metal cluster.
FYI, the fix is in 1.7.6.


Question: <ins> /* Copyright 2024 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package storage import ( "context" "fmt" "math/rand" "time" storagev1 "k8s.io/api/storage/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" "k8s.io/apimachinery/pkg/types" utilrand "k8s.io/apimachinery/pkg/util/rand" "k8s.io/client-go/util/retry" "k8s.io/kubernetes/test/e2e/framework" "k8s.io/kubernetes/test/e2e/storage/utils" "github.com/onsi/ginkgo/v2" "github.com/onsi/gomega" ) var _ = utils.SIGDescribe("VolumeAttachment", func() { f := framework.NewDefaultFramework("volumeattachment") ginkgo.Describe("Conformance", func() { ginkgo.It("should run through the lifecycle of a VolumeAttachment", func(ctx context.Context) { vaClient := f.ClientSet.StorageV1().VolumeAttachments() firstVA, vaNodeName := createVolumeAttachment(f, ctx) ginkgo.By(fmt.Sprintf("Get VolumeAttachment %q on node %q", firstVA, vaNodeName)) retrievedVA, err := vaClient.Get(ctx, firstVA, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get VolumeAttachment %q", firstVA) gomega.Expect(retrievedVA.Name).To(gomega.Equal(firstVA), "Checking that retrieved VolumeAttachment has the correct name") ginkgo.By(fmt.Sprintf("Patch VolumeAttachment %q on node %q", firstVA, vaNodeName)) payload := "{"metadata":{"labels":{"" + retrievedVA.Name + "":"patched"}}}" patchedVA, err := vaClient.Patch(ctx, retrievedVA.Name, types.MergePatchType, []byte(payload), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch PV %q", firstVA) gomega.Expect(patchedVA.Labels).To(gomega.HaveKeyWithValue(patchedVA.Name, "patched"), "Checking that patched label has been applied") patchedSelector := labels.Set{patchedVA.Name: "patched"}.AsSelector().String() ginkgo.By(fmt.Sprintf("List VolumeAttachments with %q label", patchedSelector)) vaList, err := vaClient.List(ctx, metav1.ListOptions{LabelSelector: patchedSelector}) framework.ExpectNoError(err, "failed to list VolumeAttachments") gomega.Expect(vaList.Items).To(gomega.HaveLen(1)) ginkgo.By(fmt.Sprintf("Delete VolumeAttachment %q on node %q", firstVA, vaNodeName)) err = vaClient.Delete(ctx, firstVA, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete VolumeAttachment %q", firstVA) ginkgo.By(fmt.Sprintf("Confirm deletion of VolumeAttachment %q on node %q", firstVA, vaNodeName)) type state struct { VolumeAttachments []storagev1.VolumeAttachment } err = framework.Gomega().Eventually(ctx, framework.HandleRetry(func(ctx context.Context) (*state, error) { vaList, err := vaClient.List(ctx, metav1.ListOptions{LabelSelector: patchedSelector}) if err != nil { return nil, fmt.Errorf("failed to list VolumeAttachment: %w", err) } return &state{ VolumeAttachments: vaList.Items, }, nil })).WithTimeout(30 * time.Second).Should(framework.MakeMatcher(func(s *state) (func() string, error) { if len(s.VolumeAttachments) == 0 { return nil, nil } return func() string { return fmt.Sprintf("Expected VolumeAttachment to be deleted, found %q", s.VolumeAttachments[0].Name) }, nil })) framework.ExpectNoError(err, "Timeout while waiting to confirm VolumeAttachment %q deletion", firstVA) secondVA, vaNodeName := createVolumeAttachment(f, ctx) updatedLabel := map[string]string{"va-e2e": "updated"} updatedSelector := labels.Set{"va-e2e": "updated"}.AsSelector().String() ginkgo.By(fmt.Sprintf("Update the VolumeAttachment %q on node %q with label %q", secondVA, vaNodeName, updatedSelector)) var updatedVA *storagev1.VolumeAttachment err = retry.RetryOnConflict(retry.DefaultRetry, func() error { currentVA, err := vaClient.Get(ctx, secondVA, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get VolumeAttachment %q", patchedVA.Name) currentVA.Labels = updatedLabel updatedVA, err = vaClient.Update(ctx, currentVA, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to update VolumeAttachment %q on node %q", secondVA, vaNodeName) gomega.Expect(updatedVA.Labels).To(gomega.HaveKeyWithValue("va-e2e", "updated"), "Checking that updated label has been applied") thirdVA, vaNodeName := createVolumeAttachment(f, ctx) ginkgo.By(fmt.Sprintf("Update the VolumeAttachment %q on node %q with label %q", thirdVA, vaNodeName, updatedSelector)) err = retry.RetryOnConflict(retry.DefaultRetry, func() error { currentVA, err := vaClient.Get(ctx, thirdVA, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to get VolumeAttachment %q", patchedVA.Name) currentVA.Labels = updatedLabel updatedVA, err = vaClient.Update(ctx, currentVA, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to update VolumeAttachment %q on node %q", thirdVA, vaNodeName) gomega.Expect(updatedVA.Labels).To(gomega.HaveKeyWithValue("va-e2e", "updated"), "Checking that updated label has been applied") ginkgo.By(fmt.Sprintf("DeleteCollection of VolumeAttachments with %q label", updatedSelector)) err = vaClient.DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: updatedSelector}) framework.ExpectNoError(err, "failed to delete VolumeAttachment collection") ginkgo.By(fmt.Sprintf("Confirm deleteCollection of VolumeAttachments with %q label", updatedSelector)) err = framework.Gomega().Eventually(ctx, framework.HandleRetry(func(ctx context.Context) (*state, error) { vaList, err := vaClient.List(ctx, metav1.ListOptions{LabelSelector: updatedSelector}) if err != nil { return nil, fmt.Errorf("failed to list VolumeAttachment: %w", err) } return &state{ VolumeAttachments: vaList.Items, }, nil })).WithTimeout(30 * time.Second).Should(framework.MakeMatcher(func(s *state) (func() string, error) { if len(s.VolumeAttachments) == 0 { return nil, nil } return func() string { list := []string{} for _, va := range s.VolumeAttachments { list = append(list, va.Name) } return fmt.Sprintf("Expected VolumeAttachment(s) to be deleted, found %v", list) }, nil })) framework.ExpectNoError(err, "Timeout while waiting to confirm deletion of all VolumeAttachments") }) }) }) func NewVolumeAttachment(vaName, pvName, nodeName string, status bool) *storagev1.VolumeAttachment { return &storagev1.VolumeAttachment{ ObjectMeta: metav1.ObjectMeta{ UID:  types.UID(vaName), Name: vaName, }, Spec: storagev1.VolumeAttachmentSpec{ Attacher: "e2e-test.storage.k8s.io", NodeName: nodeName, Source: storagev1.VolumeAttachmentSource{ PersistentVolumeName: &pvName, }, }, Status: storagev1.VolumeAttachmentStatus{ Attached: status, }, } } func createVolumeAttachment(f *framework.Framework, ctx context.Context) (string, string) { randUID := "e2e-" + utilrand.String(5) vaName := "va-" + randUID pvName := "pv-" + randUID nodes, err := f.ClientSet.CoreV1().Nodes().List(ctx, metav1.ListOptions{}) framework.ExpectNoError(err, "failed to list nodes") randNode := rand.Intn(len(nodes.Items)) vaNodeName := nodes.Items[randNode].Name vaAttachStatus := false ginkgo.By(fmt.Sprintf("Create VolumeAttachment %q on node %q", vaName, vaNodeName)) va := NewVolumeAttachment(vaName, pvName, vaNodeName, vaAttachStatus) createdVA, err := f.ClientSet.StorageV1().VolumeAttachments().Create(ctx, va, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create VolumeAttachment %q", vaName) gomega.Expect(createdVA.Name).To(gomega.Equal(vaName), "Checking that the created VolumeAttachment has the correct name") return createdVA.Name, vaNodeName } </ins>

[x] two weeks soak end date : 14 Mar 2024 : According to following APIsnoop query, there are six VolumeAttachment endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following VolumeAttachment endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } } <ins> func TestIdleShutdownRace(t *testing.T) { var wg sync.WaitGroup server, listen, serverErr := runServer(&wg) if serverErr != nil { t.Fatalf("Error initializing server: %s", serverErr) } conn, dialErr := net.Dial("tcp", listen) if dialErr != nil { t.Fatalf("Error dialing server: %s", dialErr) } spdyConn, spdyErr := NewConnection(conn, false) if spdyErr != nil { t.Fatalf("Error creating spdy connection: %s", spdyErr) } go spdyConn.Serve(NoOpStreamHandler) authenticated = true stream, err := spdyConn.CreateStream(http.Header{}, nil, false) if err != nil { t.Fatalf("Error creating stream: %v", err) } spdyConn.SetIdleTimeout(5 * time.Millisecond) go func() { time.Sleep(5 * time.Millisecond) stream.Reset() }() select { case <-spdyConn.CloseChan(): case <-time.After(20 * time.Millisecond): t.Fatal("Timed out waiting for idle connection closure") } closeErr := server.Close() if closeErr != nil { t.Fatalf("Error shutting down server: %s", closeErr) } wg.Wait() } </ins> func TestIdleNoTimeoutSet(t *testing.T) { var wg sync.WaitGroup server, listen, serverErr := runServer(&wg)

Hit this while :
cc/ Andy can you take a look at this failure?
looking now
Working with on a fix
Proposed fix:


Question: gomega.Expect(c.CoreV1().Pods(ns).Delete(context.TODO(), "run-test", metav1.DeleteOptions{})).To(gomega.BeNil()) ginkgo.By("executing a command with run and attach without stdin") <del> runOutput = framework.NewKubectlCommand(ns, fmt.Sprintf("--namespace=%v", ns), "run", "run-test-2", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--leave-stdin-open=true", "--", "sh", "-c", "cat && echo 'stdin closed'"). </del> <ins> // There is a race on this scenario described in #73099 // It fails if we are not able to attach before the container prints // "stdin closed", but hasn't exited yet. // We wait 5 seconds before printing to give time to kubectl to attach // to the container, this does not solve the race though. runOutput = framework.NewKubectlCommand(ns, fmt.Sprintf("--namespace=%v", ns), "run", "run-test-2", "--image="+busyboxImage, "--restart=OnFailure", "--attach=true", "--leave-stdin-open=true", "--", "sh", "-c", "sleep 5; cat && echo 'stdin closed'"). </ins> WithStdinData("abcd1234"). ExecOrDie(ns) gomega.Expect(runOutput).ToNot(gomega.ContainSubstring("abcd1234"))

The test case "executing a command with run and attach without stdin" is inherently racy. The test runs, and expected to see in the output. The case when the test passes: case 1: Kubernetes creates and starts the container without stdin; The container exits before attach runs; kubectl falls back to get container log, which contains . case 2: Kubernetes creates and starts the container without stdin; kubectl attach before "stdin closed" is printed; kubectl attach contains . The case when the test fails: Kubernetes creates and starts the container without stdin; The container prints "stdin closed", but hasn't exited yet; kubectl attach contains nothing. One example failure: Related issue: Can we modify this test not to check the output? Or use to check output instead? /cc
Uploaded to try to address
/reopen This test is still inherently racy, even after
Fix in (I hope!)
Moving into the current milestone as it was not fixed in 1.14
Hello I'm a bug triage shadow for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. Code freeze is starting on August 29th (about 3 days from now). Do we still target this issue to be fixed for 1.16?
Thanks . PR is available and awaiting review!
What is the PR in question? Is it targeting 1.16 or 1.17?
/reopen This test is having a lot of failures these days
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is right, it seems to be present in kind jobs now only
/assign let's see if we start to recover a bit of stability :smile:
The rate of failures decreased a lot since last changes Checking the logs of the failures it seems that the problem is that the pod has not finished before checking the output, as Lantao described The test fails Collecting events from namespace The framework logs the pods the kubelet thinks is on the node
note that it's still racy, but it's hopefully somewhat mitigated.
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
any updates here?
nope, totally agree with the description, is racy by design, couldn't find any way or trick to avoid the randomness
EDIT: It's late and I'm actually not sure I'm remembering this correctly, take this comment with a heap of salt: Specific sub-tests were fixed previously I think, but this one sub-case: That one we can't use the "Wait for stdin input to signal we're attached" hack, so it's just racy and broken. We could remove that case maybe. I'm not sure who I'd want to weigh in on that, probably someone more active in node / CLI.
not super common atm but still happening. discussing
/unassign has a fix for this


Question: gvk := unstructured.GetObjectKind().GroupVersionKind() t := c.parser.Type(gvk) if t == nil { <del> return nil, fmt.Errorf("no corresponding type for %v", gvk) </del> <ins> return nil, newNoCorrespondingTypeError(gvk) </ins> } return t.FromYAML(typed.YAMLObject(string(from))) }

Found by failing TestApplyConvertsManagedFieldsVersion test in () It seems that conversion depends on schema info that is collected from served API versions only. However, managedFields stores data in etcd from REST versions served in previous releases which may no longer be served. That means that we have to be able to deal with managedFields persisted via API calls to versions that are no longer served, but which we still have and understand conversions for. /cc /sig api-machinery /wg apply /priority important-soon
This may be fixed in a more recent version of structured-merge-diff. I'll look into it
/assign
At one point we were using the types registered to the scheme to generate the openAPI definitions used by apply, but we switched to using the installed endpoints instead, because it was less invasive (We needed to expose which types are convertible to which other types, and also a way to skip generating openAPI for internal types). We might want to revisit the first approach, given this use case.


Question: diskName, diskURI, requestGB, err := b.CreateVolume("testBlob", "testsa", "type", b.common.location, 10) expectedErr := fmt.Errorf("could not get storage key for storage account testsa: could not get storage key for "+ "storage account testsa: Retriable: false, RetryAfter: 0s, HTTPStatusCode: 500, RawError: %w", error(nil)) <del> assert.Equal(t, expectedErr, err) </del> <ins> assert.EqualError(t, err, expectedErr.Error()) </ins> assert.Empty(t, diskName) assert.Empty(t, diskURI) assert.Zero(t, requestGB)

Our internal CI job that runs k8s unit tests with master golang. TestCreateVolume TestDeleteVolume Probably after the master golang GITCOMMIT= Change: https://go- No response No response /sig testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
^^
Updates: Observed that with the latest version of Go dev branch, the underlying type is no longer fmt.WrapError, but is rather errors.ErrorString when a nil error is using the verb is used. Both, the expected error and the returned error had a similar type until go 1.19 - , which now switched to when a error is wrapped with the verb. The changes are made in where the earlier used is now replaced with to check for error equality. The changes have been tested with both go 1.19 and go 1.20-devel and works as intended.


Question: }, }, { <ins> name: "handles watch delete multiple", infos: []*resource.Info{ { Mapping: &meta.RESTMapping{ Resource: schema.GroupVersionResource{Group: "group", Version: "version", Resource: "theresource-1"}, }, Name:      "name-foo-1", Namespace: "ns-foo", }, { Mapping: &meta.RESTMapping{ Resource: schema.GroupVersionResource{Group: "group", Version: "version", Resource: "theresource-2"}, }, Name:      "name-foo-2", Namespace: "ns-foo", }, }, fakeClient: func() *dynamicfakeclient.FakeDynamicClient { fakeClient := dynamicfakeclient.NewSimpleDynamicClient(scheme) fakeClient.PrependReactor("get", "theresource-1", func(action clienttesting.Action) (handled bool, ret runtime.Object, err error) { return true, newUnstructured("group/version", "TheKind", "ns-foo", "name-foo-1"), nil }) fakeClient.PrependReactor("get", "theresource-2", func(action clienttesting.Action) (handled bool, ret runtime.Object, err error) { return true, newUnstructured("group/version", "TheKind", "ns-foo", "name-foo-2"), nil }) fakeClient.PrependWatchReactor("theresource-1", func(action clienttesting.Action) (handled bool, ret watch.Interface, err error) { fakeWatch := watch.NewRaceFreeFake() fakeWatch.Action(watch.Deleted, newUnstructured("group/version", "TheKind", "ns-foo", "name-foo-1")) return true, fakeWatch, nil }) fakeClient.PrependWatchReactor("theresource-2", func(action clienttesting.Action) (handled bool, ret watch.Interface, err error) { fakeWatch := watch.NewRaceFreeFake() fakeWatch.Action(watch.Deleted, newUnstructured("group/version", "TheKind", "ns-foo", "name-foo-2")) return true, fakeWatch, nil }) return fakeClient }, timeout: 10 * time.Second, validateActions: func(t *testing.T, actions []clienttesting.Action) { if len(actions) != 2 { t.Fatal(spew.Sdump(actions)) } if !actions[0].Matches("list", "theresource-1") { t.Error(spew.Sdump(actions)) } if !actions[1].Matches("list", "theresource-2") { t.Error(spew.Sdump(actions)) } }, }, { </ins> name: "ignores watch error", infos: []*resource.Info{ {

We should consider updating the command to wait on all resources for a given kind. For example: would wait for every pod in the current namespace to be deleted. /sig cli cc
Interestingly, the error text implies that this is already supported: Yet when you try to use it, it declares Would really like the wait functionality given some much needed TLC. Still no way to determine what conditions are supported for various resources (eg. I can say and it just waits for the timeout like that resource might actually potato at some point).
I'm looking into adding the --all flag to wait, sounds like a nice feature to have :)


Question: glog.V(4).Infof("Ensure an external loadbalancer service.") internalAnnotation = false } else { <del> return nil, fmt.Errorf("floating-network-id or loadbalancer.openstack.org/floating-network-id should be specified when service.beta.kubernetes.io/openstack-internal-load-balancer is false") </del> <ins> return nil, fmt.Errorf("floating-network-id or loadbalancer.openstack.org/floating-network-id should be specified when ensuring an external loadbalancer service.") </ins> } default: return nil, fmt.Errorf("unknow service.beta.kubernetes.io/openstack-internal-load-balancer annotation: %v, specify "true" or "false".",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Creating a service with created a load balancer without a floating IP. What you expected to happen: Load balancer should have a floating IP assigned by default. How to reproduce it (as minimally and precisely as possible): Deploy this manifest on a cluster and check the resulting load balancer. No floating IP will be assigned. Anything else we need to know?: The default behavior changed from k8s 1.7.x. It appears that an annotation that will add a floating IP, but the default should not have changed without warning. This also breaks the workflow. For example: Environment: Kubernetes version (use ): v1.8.0-rc.1+coreos.0 Cloud provider or hardware configuration**: openstack OS (e.g. from /etc/os-release): Container Linux by CoreOS 1520.4.0 (Ladybug) Kernel (e.g. ): 4.13.3-coreos Install tools: Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig openstack
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Oops, default should be external. /assign


Question: serv.Flags |= utilipvs.FlagPersistent serv.Timeout = uint32(svcInfo.StickyMaxAgeSeconds()) } <ins> // Set the source hash flag needed for the distribution method "mh" if proxier.ipvsScheduler == "mh" { serv.Flags |= utilipvs.FlagSourceHash } </ins> // We need to bind ClusterIP to dummy interface, so set `bindAddr` parameter to `true` in syncService() if err := proxier.syncService(svcPortNameString, serv, true, bindedAddresses); err == nil { activeIPVSServices[serv.String()] = true

The current support of the load distribution method and in kube-proxy ipvs mode only performs hashing of source IP address to select endpoints. As a result, all connections from a single client (i.e. from a particular source IP) are always forwarded to only one endpoint matching with the hash value of this source IP. If the client opens a vast majority of connections to the k8s cluster, it will result in traffic imbalance as only one endpoint located on one node handles the connections while the remaining endpoints/nodes are idle. Here is an example of the problem when using method for kube-proxy Deploy a service with 4 endpoints located on 4 nodes.Open 200 connections from a client to node IP of any node (e.g. telnet 192.168.1.1 ). All connections receive replies from only one single endpoint.The distribution printout also shows that all connections are forwarded to the endpoint (e.g. 1.0.4.2:5001 in this example)Similar issue is seen with method Connections from a client should be evenly distributed to all endpoints like to avoid unbalanced load distribution between cluster nodes and endpoints. This can be easily done by enabling the source port hash flag that IPVS supports for the methods which performs hashing on both source IP and source port when selecting the target endpoint.
/sig network /area kube-proxy /area ipvs
This issue was raised based on the discussion of the issue
triage/accepted /assign
/triage/accepted
/triage accepted (damn it!)


Question: import ( "os" <ins> "time" "k8s.io/apiserver/pkg/server" </ins> "k8s.io/klog" "golang.org/x/sys/windows"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I configured Kubelet to run as a service on Windows, with a setting to restart on failure (). I noticed that it restarts even when I explicitly stop the service with , even though the documentation states the following (I left it at , the default): What you expected to happen: I expected the Kubelet to only restart when killed unexpectedly, per above docs. Anything else we need to know?: I think the problem might be due to this part of our service control integration, but I'm not really sure: It looks like we send and then , but the official golang example sends and just returns without exiting. (). Is this difference meaningful? Kubernetes version (use ): 1.13.2 Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Windows /cc /sig node /sig windows
The same should apply to kube-proxy as well.
i believe you implemented this functionality. can you please take a look? cc:


Question: return *machine.ID, nil } <ins> func (as *availabilitySet) GetProvisioningStateByNodeName(name string) (provisioningState string, err error) { vm, err := as.getVirtualMachine(types.NodeName(name)) if err != nil { return provisioningState, err } return *vm.ProvisioningState, nil } </ins> // GetNodeNameByProviderID gets the node name by provider ID. func (as *availabilitySet) GetNodeNameByProviderID(providerID string) (types.NodeName, error) { // NodeName is part of providerID for standard instances.

Implement proposal for Azure We should implement interface in Azure cloud provider. returns true if the instance is in stopped state to detach volumes. But if instance is deleted we should not return error. Related features: Related proposal: Related PRs: /kind feature /sig azure /cc
- can you assign me on this one ? I already have a PR in flight.
That's great! /assign just loop in for this RP review, thanks.
GitHub didn't allow me to assign the following users: yastij. Note that only and repo collaborators can be assigned. <detailsIn response to : That's great! /assign just loop in for this RP review, thanks. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
o, you are not a k8s member yet, anyway, you could send out the PR for this feature, related to this issue, thanks.
Thanks, looking forward to your PR.


Question: os.Exit(1) } <del> func connectSCTP(dest string, timeout time.Duration) { </del> <ins> func connectSCTP(dest string, timeout time.Duration, data string) { var ( buf  = make([]byte, 1024) conn *sctp.SCTPConn ) </ins> addr, err := sctp.ResolveSCTPAddr("sctp", dest) if err != nil { fmt.Fprintf(os.Stderr, "DNS: %vn", err)

How to repro: an agnhost pod listening on sctp into the pod and use the connect command to probe against the same pod The stack trace that points to and this is from the vendored library , /kind bug /sig network /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Run an agnhost pod listening on sctpExec into the pod and use the connect command to probe against the same pod The stack trace that points to /sig network /help </details
/assign
/triage accepted
here we are immediately closing the connection after successful connect leading to EOF at server end (it is expecting data) causing the failure. We can tweak this to connect send some data just like UDP and then exit? Works locally writing "hostname" to the connection. [EDIT] For me it sometimes failed with EOF, sometime with nil pointer exception at . This reads sock opts via syscall and we immediately close the connection .. kind of a race between the client closing connection and server reading fds?
the panic seems a bug in that project, no? Also SCTP, has a establishment handshake, so the data should not be relevant in this case
returns fd and socket address, for is used to implement (always read from struct field). The SCTP implementation discards the returned socket address by call. , and is always a new syscall (). In our case the client immediately closes the connection, there can be chances where SCTPGETPEERADDRS will fail and will return nil in these cases. I think this is reasonable for their project. We call and panic is raised by calling on nil object. We can easily avoid this by not null check.

Also, I strongly believe that the client handling should be done in a separate go-routine instead of the main loop to avoid errors like this crashing the pod. [EDIT] The client in our case needs to send some data after connection, or else the server will throw EOF and we currently have assertions that use for errors.
panicing does not seem very reasonable It looks to me that the problem is that the SCTPConn struct needs to check if the connection is still valid before trying to access the file descriptor , see the std library implementation and then, all methods implement this check to me it is a bug on the implementation
The implementation is not causing any panic, it simply returns nil. Panic is caused when we try to call on nil object.
oh, sorry, my bad I should read your comment with more detail, is well explained there I recognize I have zero idea of how sctp works internally, maybe we can just be more resilient on our tests and code and detect if is nil and avoid getting into the complexity of this protocol ... sending some random data also sounds good, we do this for udp IIRC
are you able to send a PR to add those improvements to the test image?


Question: testEndpointReachability(clusterIP, sp.Port, sp.Protocol, execPod) } } func testReachabilityOverNodePorts(nodes *v1.NodeList, sp v1.ServicePort, pod *v1.Pod) { internalAddrs := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) externalAddrs := e2enode.CollectAddresses(nodes, v1.NodeExternalIP) for _, internalAddr := range internalAddrs { <ins> // If the node's internal address points to localhost, then we are not // able to test the service reachability via that address if isInvalidOrLocalhostAddress(internalAddr) { e2elog.Logf("skipping testEndpointReachability() for internal adddress %s", internalAddr) continue } </ins> testEndpointReachability(internalAddr, sp.NodePort, sp.Protocol, pod) } for _, externalAddr := range externalAddrs {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: CRI-O e2e Which test(s) are failing: e2e suite [sig-network] Services should be able to change the type from ExternalName to NodePort e2e suite [sig-network] Services should be able to create a functioning NodePort service 19s See link: https://openshift-gce- Since when has it been failing: Anything else we need to know:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/milestone v1.16 /assign FYI Looks like two other release-blocking tests too: - The changes link testgrid gives () suggests two other PR's, but I agree that seems the likely culprit
Based on the , Service is reachable on and but failing to reach on node's IP over nodePort i.e. . From the log, it is not clear why its not reachable on node's internal IP. Could you provide from the CRI-O based node when trying to reach nodePort service. Secondly, need to make sure, node is listening on allocated nodePort. As this issue seems much on system-level, I need to setup k8s cluster/node with CRI-O runtime and troubleshoot to find the root cause.
[sig-network] ESIPP [Slow] [DisabledForLargeClusters] should work from pods [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] seems flaky to me. Most of time execPod starts running and might be terminating immediately after some time which causes this issue. [1.] e2e is failing as might have terminated and so no PodIP leading to invalid . [2.] empty . The same issue had been observed in for E2E and to fix the issue, we introduced execPod deployment to make sure execPod would remain in desired running state. It's better to address this change in another PR and emphasize using pausePod deployment.
Thanks, for helping me setting up CRI-O based cluster.

Kubernetes e2e suite [sig-network] Services should be able to change the type from ExternalName to NodePort e2e suite [sig-network] Services should be able to create a functioning NodePort service 19s From log, it is clear that, these services are failing to reach over allocated nodePort. But on my CRIO based testing cluster, they are reachable. or Could you please check if they failing at your end too or it's testgrid Job-specific issue?
Hey, I did some local tests and was able to reproduce the failure. I started the cluster via (check that the variable is correct and does not point to ): Executed the tests by (again, watch out for the variable to be correct): Error output: Since I'm now able to reproduce the issue I can go further in debugging and find out how to fix this. Is anyone furhter, or If not, then I will target for fixing the issue this week. :)
Thanks for reproducing this. FYI we've seem similar failure on , but I haven't reproduced the problem locally yet. We are looking to at least turn on verbose on the test command for debugging:
The issue is that we try to automatically discover the node IP by obtaining its status here: In our setup the node has IP which for sure does not work if we try to access it within the pod. I'm thinking about a proper solution and will propose it as a PR.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/assign