Question: } } } <ins> func TestMaxUnavailable(t *testing.T) { deployment := func(replicas int32, maxUnavailable intstr.IntOrString) extensions.Deployment { return extensions.Deployment{ Spec: extensions.DeploymentSpec{ Replicas: func(i int32) *int32 { return &i }(replicas), Strategy: extensions.DeploymentStrategy{ RollingUpdate: &extensions.RollingUpdateDeployment{ MaxSurge:       func(i int) *intstr.IntOrString { x := intstr.FromInt(i); return &x }(int(1)), MaxUnavailable: &maxUnavailable, }, Type: extensions.RollingUpdateDeploymentStrategyType, }, }, } } tests := []struct { name       string deployment extensions.Deployment expected   int32 }{ { name:       "maxUnavailable less than replicas", deployment: deployment(10, intstr.FromInt(5)), expected:   int32(5), }, { name:       "maxUnavailable equal replicas", deployment: deployment(10, intstr.FromInt(10)), expected:   int32(10), }, { name:       "maxUnavailable greater than replicas", deployment: deployment(5, intstr.FromInt(10)), expected:   int32(5), }, { name:       "maxUnavailable with replicas is 0", deployment: deployment(0, intstr.FromInt(10)), expected:   int32(0), }, { name: "maxUnavailable with Recreate deployment strategy", deployment: extensions.Deployment{ Spec: extensions.DeploymentSpec{ Strategy: extensions.DeploymentStrategy{ Type: extensions.RecreateDeploymentStrategyType, }, }, }, expected: int32(0), }, { name:       "maxUnavailable less than replicas with percents", deployment: deployment(10, intstr.FromString("50%")), expected:   int32(5), }, { name:       "maxUnavailable equal replicas with percents", deployment: deployment(10, intstr.FromString("100%")), expected:   int32(10), }, { name:       "maxUnavailable greater than replicas with percents", deployment: deployment(5, intstr.FromString("100%")), expected:   int32(5), }, } for _, test := range tests { t.Log(test.name) maxUnavailable := MaxUnavailable(test.deployment) if test.expected != maxUnavailable { t.Fatalf("expected:%v, got:%v", test.expected, maxUnavailable) } } } </ins>

We need to gracefully handle the case when a Deployment specifies an absolute value of MaxUnavailable larger than its spec.replicas. I don't think we want to add validation for this, simply collapsing maxUnavailable to spec.replicas would suffice.
Are you working on this? I am glad to take this if not.
feel free to work on it


Question: }() } wg.Wait() <del> return atomic.LoadInt32(&completed) </del> <ins> return completed </ins> }

Should we create tests around the creation, modification and deletion of the objects ? For example, adding an invalid FlowSchema with a misconfigured PriorityLevelConfiguration and verifying that the Status appropriately reflects its invalidity. I’m assuming the controller tests will suffice and we don’t need explicit e2e tests for this. What about CRUD tests wherein we’re validating the debug endpoints are showing the right values ? The first class of tests are the basic ones to verify that the requests are being categorized into the right FlowSchema and Priority level. We’ll set up a new PriorityLevelConfiguration and FlowSchema objects and observe that requests are correctly being classified onto them. Verification that requests are correctly being classified will be tricky. For now, we can scrape the metrics endpoint for ‘’ metric to verify that the expected number of requests for a test FlowSchema is correct. Another option is to utilize the ‘dump_requests’ debug endpoint to verify the correct request classification. If during the course of writing the tests we need additional information, we can extend the debug endpoint. Scraping the metrics endpoints to validate the proper request categorization could lead to a fickle e2e test. So querying the ‘debug’ endpoint seems like the right way to go. This test will also serve as a proxy for an end-to-end test that ensures that requests matching a new FlowSchema are successful. Does this have the propensity to be a flaky test(s) ? This class of tests will focus on validating the behavior this feature promises, viz., requests of low priority won’t be drowned out by requests of higher priority and a single bad actor won’t detrimentally affect the requests from others. The following tests will ensure that high velocity flows don’t affect the low velocity flows. For this test, we’ll have two entities submitting requests at the same time. These requests will map to two different FlowSchemas at different priority levels. The higher priority flow will have a lot of requests going through it, far above its concurrency share. At the same time the low priority flow will be submitting requests as well at a rate that well within the concurrency limits of that Flow. The test will ensure that the low velocity (low priority) flow succeeds. This test will be the same as the above one except that the two flows will map to the same FlowSchema but are separated via their Flow Distinguisher.

One of them will be a high velocity flow which the other is a low velocity one. The test will ensure that the low velocity requests still get through and succeed. Does this have the propensity to be a flaky tests ? If the test fails, will the runner of this test be able to grok where the problem could lie and debug ? Is this an actionable failure ?
cc/
We don't need to test that invalid items are rejected, unit tests are better for that. We should try to exercise the basic features (exempt, queue...) We should not rely on the debug endpoint, since we want this to be a conformance test eventually Metrics scraping is also a little questionable. I think we should exercise fair queueing (make two new PLs at different shares, saturate both, verify the responses follow the share ratio). I think we should exercise the shuffle sharding (make a new PL, saturate it with one/few elephants and many mice, verify the mice get service. (those latter two seem to match your last two.) This would be a pretty high level test that tests things end users will care about, but I think it should still be quite hard to pass without actually running our code.
/sig api-machinery /area test /kind feature
cc
to verify the requests can be correctly classified, how about we make use of the ? we can do a test like: make two PLs, and two clients w/ different identities then make the clients are hitting different FS/PL.
what about using a test aggregated apiserver to serve different PLs, and with different the response latency
No strong opinion on this one. I am not sure if that was a response to the ideas in the "Basic CRUD" section --- there the idea is not about testing validation but rather testing the status setting about referential integrity. The debug endpoints are about what is happening at the moment of the GET. Controlling that tightly enough to be worth a test would be tricky. The only plausible approach in an E2E test would be to register an authorization webhook that synchronizes with the test function. I like the basic idea. For the reasons above, I would not recommend looking at the debug endpoints. I do not understand what is wrong with scraping the metrics. Are they not part of conformance? This feature does add response headers.

Are those part of conformance? I like the basic idea. First, remember that this feature does not control velocity; it controls the product of velocity and execution duration. To get a high signal-to-noise ratio, we would want deterministic and large execution durations. Looking in , you can see that authorization is included in execution. So we could add a controlled large amount of duration in there. I am a little surprised at the question of "If the test fails, will the runner of this test be able to grok where the problem could lie and debug"; why would that be more of a question here than for any other E2E test? Presumably the test failure message will say something about the expected and observed behavior. I think the drown-out scenario involves testing queuing. I think exempt was not covered above, and would be a worthwhile addition. This mixes up two things. Fair queuing is something that happens within one priority level. This sounds like it may be expensive for an E2E test. I think we can make a much more efficient unit test of QueueSet that exercises shuffle sharding. Because the QueueSet implementation can be given a FakeEventClock, we can run the test much faster than real time. To get a high signal-to-noise ratio we want controlled, high latencies. Making an aggregated apiserver do that is possible. Perhaps the simplest approach would be to put a fixed sleep in the registry strategies for whatever custom resources the aggregated apiserver is serving. But this seems like a lot more work than necessary. I think an authorization webhook is a simpler approach.
could you elaborate a bit more about the authorization webhook approach? it is only called in the authorization path, if i understand it correctly.
Authorization, and admission control, happen after the priority and fairness filter.
is this really possible? I am likely missing something, but my understanding from the is that each level is limited to the concurrency shares assigned to it.
: the text should be revised to more clearly express the intent. It is that the driving client has many more requests outstanding at once than the corresponding priority level will allow to be served concurrently.
FYI: I'm working on the second test.


Question: if delay > 0 { glog.Warningf("Inserting delay before AWS request (%s) to avoid RequestLimitExceeded: %s", describeRequest(r), delay.String()) <del> r.Config.SleepDelay(delay) </del> <ins> if sleepFn := r.Config.SleepDelay; sleepFn != nil { // Support SleepDelay for backwards compatibility sleepFn(delay) } else if err := aws.SleepWithContext(r.Context(), delay); err != nil { r.Error = awserr.New(request.CanceledErrorCode, "request context canceled", err) r.Retryable = aws.Bool(false) return } </ins> // Avoid clock skew problems r.Time = now


Question: klog.Warningf("parseTags: error when parsing key-value pair %s-%s, would ignore this one", k, v) continue } <del> formatted[k] = to.StringPtr(v) </del> <ins> formatted[strings.ToLower(k)] = to.StringPtr(v) </ins> } return formatted }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- " TestEnsurePIPTagged/ensurePIPTaggedshouldensurethepipistaggedasconfigured": 8, " TestEnsurePIPTagged": 8, failed exmaple links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider /cc as the test case is updated recently in .
flaking badly since the merged update -
/assign
to fix


Question: } } allErrs = append(allErrs, validateInitializersResult(initializers.Result, fldPath.Child("result"))...) <del> if len(initializers.Pending) == 0 && initializers.Result == nil { allErrs = append(allErrs, field.Invalid(fldPath.Child("pending"), nil, "must be non-empty when result is not set")) } </del> return allErrs }

There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery


Question: return parsed } <del> // fmtHTML parses and re-emits 'in', effectively canonicalizing it. func fmtHTML(in string) string { doc, err := html.Parse(strings.NewReader(in)) if err != nil { panic(err) } out := &bytes.Buffer{} if err := html.Render(out, doc); err != nil { panic(err) } return string(out.Bytes()) } </del> func TestProxyTransport(t *testing.T) { testTransport := &Transport{ Scheme:      "http",

The ApiServer proxy wraps partial HTML responses to serve a complete HTML document in the proxy response. This is problematic for applications which expect HTML fragments, such as with AngularJS templates. Example: This is the response for a from the unproxied server: But the proxied response is: This issue is blocking


Question: createTagFactor       = 2.0 createTagSteps        = 9 <ins> // encryptedCheck* is configuration of poll for created volume to check // it has not been silently removed by AWS. // On a random AWS account (shared among several developers) it took 4s on // average. encryptedCheckInterval = 1 * time.Second encryptedCheckTimeout  = 30 * time.Second </ins> // Number of node names that can be added to a filter. The AWS limit is 200 // but we are using a lower limit on purpose filterNodeLimit = 150

This is continuation of and . What happened: When StorageClass refers to a key that is not accessible to Kubernetes (i.e. AWS API returns AccessDeniedException), AWS plugin still tries to provision a volume under assumption that while Kubernetes can't access the key, the other components may access it and create the volume and attach it to a node later. Related code is - checks only for . Other errors are just logged and provisioning continues. Problem is that when AWS really can't access the key, it will provision some volume (ec2.CreateVolume returns success + valid AWS EBS), but this AWS EBS disappears after few seconds. Therefore Kubernetes does not know that something went wrong and creates a PV for the returned EBS. Kubernetes later fails to run a pod that refers to such PV, because the PV does not exist. What you expected to happen: Kubernetes must not create PV in this case and must report error during provisioning of such volume. Ideas: According to , the only signal that AWS produces is an event in Amazon CloudWatch Events that a key could not be used for encryption. Kubernetes AWS cloud provider is not subscribed to these events and does not get this notification. We would need new permission for controller-manager to get the events and some new code to filter them and report back to user. Kubernetes AWS cloud provider could wait for X seconds and re-check that created volume still exists and did not disappear because of inaccessible key. What would be reliable value of X? Kubernetes volume plugin for AWS could return both a warning "I failed to check encryption key, created volume may not work" that would be propagated to user and the suspicious volume at the same time, but the API between volume plugins and Kubernetes allows only one - either error or volume. It could be extended, still the user experience would be suboptimal. Kubernetes could send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions. Again, UX is quite bad here. /kind bug /sig aws
Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. If the main source of errors are sent via CloudWatch Events, then best practice would seem to be to subscribe via SNS to receive those events.

Possibly fall back to vague "send an error to user during volume attach (!) that a volume does not exist, please check your KMS key permissions," where a SNS topic isn't provided, but otherwise use CloudWatch events to enrich state/messages. Would then also need to change docs to reflect needing to set up CloudWatch Events -SNS topic. Big drawback of going down this route is a fair few downstream cluster bootstrap projects would likely need updating to support it in their CloudFormation / Terraform bits.
According to it takes quite a long time until an event is available in CloudWatch events. what was the average / maximum delay?
I haven't calculated the metrics accurately but the cloudwatch events arrival vary(and standard deviation was high). Using cloudwatch events was also considered for maintaining instance cache and as well concluded that - they are not realtime (AWS itself documents them as "near realtime"). It may be - we need to measure them again and find out if we can use them for the purpose here. Checking the KMS key feels wrong here, as you could still create an encrypted EBS volume and not have the DescribeKey permission in IAM or via the key policy. from jan's analysis above it appears that - if IAM does not have permission to DescribeKey then EBS volume gets created but then it disappears soon. So net result still is - if "current user" can't describekey, she can't provision the volume. Is there any documentation that indicates otherwise? cc
Testing this for real now. User creates volume with no access to underlying CMK, or insufficient permissions does not seem to generate events with sufficient information as to what went wrong, i.e. don't appear to receive events for create volume failure, only deletion. E.g.: This policy is sufficient to allow the creation of volumes - isn't necessary for the volume to be created. It isn't sufficient to attach the volume however, creating the following error, The permission's required to fix this, to delegate access to EC2 itself: Bit of experimentation shows that only is required for attaching - the ones for creating the volume aren't needed at the attachment stage. The docs at is therefore correct for volume creation, but doesn't describe the volume attachment scenario.
So, what is our solution then? From the above it seems Kubernetes can't reliably check if a key is usable for encryption of a PV.

And it can't get an usable event either.
Possibly one of the following: change. Live with the pain. "GenerateDataKey" and "Decrypt" on the generated ciphertext blob prior to volume creation as a preflight check. The advantage being that we can throw an error that this isn't going to work beforehand, but with the disadvantage of doubling up API calls. This is probably fine, and I think if you're creating encrypted volumes at a rate that's likely to run into API call limits or get super expensive, you might have other problems? We then also do the CreateGrant check during volume attachment (as an optional extra). the same check, but only after an error. Problem being that we get much less in the way of guaranteeing what the root cause is, but we don't add to the API calls in the happy path. In any case, is this bug biting sufficiently to fix here, or should we start thinking about the CSI implementation?
thanks a lot for detailed investigation. I'll try to get some time to implement 2. even before CSI. Users are very innovative in setting up their cluster permissions wrong and the messaging to user is very confusing in this particular issue.
/assign
2. does not work. Kubernetes does not need to have GenerateDataKey and Decrypt permissions to create encrypted volume. My AWS account is severely limited (I am using shared company account) and I can't call GenerateDataKey on a key: But I can create encrypted EBS using the same key: The volume gets created successfully. It's not the user who runs Kubernetes who needs permissions for GenerateDataKey, it's AWS itself.
And the volume stayed created and didn't get reaped after a few seconds? I was confused about this part, it didn't seem to follow how the APIs should work. Has anyone ever created a volume with that key, or is it a new one? I'll try what I did previously in a bit across a few regions.
Yes, the volume stayed around for several minutes, then I deleted it manually. The key is our default encryption key, so when I create an encrypted volume with no key, I get this one. No zones, no regions involved.
I got an idea, the volume is "creating" and then it gets "available".

I could in a loop until newly created volume either gets "Available" or deleted. This will abuse the API quota ( is cheap, right?) + it will slow down provisioning a bit - all volumes I tried were Available in a second or two. Will it work reliably enough?
Might be the difference then, as I don't think the same perms as I described above are needed for the default key as I assume that it gets automatically delegated to the EC2 service. I only tested & documented the explicit, alternate key path above. Your approach should work, but I don't know when EC2 API limits will kick in and how soon users would see that happen. Worth a shot though?
It takes ~4 seconds for a volume to reach "available", it's quite a long time. This will affect encrypted volumes with explicit keys only.
Fix:


Question: "fmt" "net/http" "net/http/httptest" <ins> "reflect" </ins> "strconv" "testing" "time"

bug (): Originally posted by in cc /sig network /priority critical-urgent
fix in
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: maxStartAge = time.Hour * 24 * 365 // 1 year maxStatsAge = time.Minute ) <del> fsCapacityBounds := bounded(100*framework.Mb, 100*framework.Gb) </del> <ins> // fetch node so we can know proper node memory bounds for unconstrained cgroups node := getLocalNode(f) memoryCapacity := node.Status.Capacity["memory"] memoryLimit := memoryCapacity.Value() fsCapacityBounds := bounded(100*framework.Mb, 10*framework.Tb) </ins> // Expectations for system containers. sysContExpectations := func() types.GomegaMatcher { return gstruct.MatchAllFields(gstruct.Fields{

/kind bug What happened: e2e node summary test fails on machines with more than 10Gb of ram. What you expected to happen: memory limit bounds to look at node capacity rather than hardcode an upper bound. How to reproduce it (as minimally and precisely as possible): have more ram on a box and run e2e-node FOCUS=Summary Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig node


Question: return printer, nil } <del> // ClientMapperForCommand returns a ClientMapper for the given command and factory. func (f *Factory) ClientMapperForCommand(cmd *cobra.Command) resource.ClientMapper { </del> <ins> // ClientMapperForCommand returns a ClientMapper for the factory. func (f *Factory) ClientMapperForCommand() resource.ClientMapper { </ins> return resource.ClientMapperFunc(func(mapping *meta.RESTMapping) (resource.RESTClient, error) { return f.RESTClient(mapping) })

I think the need for this argument to the factory methods has passed - it means that callers who want to provide configurable behavior have to do it through the factory itself, which is acceptable. We can remove it whenever.
I removed most of the unused arguments. But, There are a few methods still using the argument. Namely, , , and . Before proceeding any further I wanted to ask what should be done about these methods? Deprecated/Refactored/Removed ?
The factory methods were the most important. The pattern before was people using the flags by name (cmd.Flags().Get("foo")), but now factory methods must use pointer flag vars, so there's nothing that should need the command there. The printer methods are optional units that can be used when you add the flags for them to your command. Unless we refactor the definition of those flags to use pointers and return a printer struct (something I had started a while back and never finished), it's not worth changing them right now. The other consumer of factory is OpenShift - we wrap the factory to add our additional types (implementations of things like describers, action verbs, etc): - but we will pick up and remove use of cmd when we rebase.
Thanks for the explanation. I've cleaned the as well. Print method APIs are untouched as you suggested.
cc
Looks like this was done by PR


Question: // kubelet on Fedora CoreOS distribution func setKubeletSELinuxLabels(host, workspace string) error { cmd := getSSHCommand(" && ", <del> fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "kubelet")), </del> <ins> fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t kubelet_exec_t %s", filepath.Join(workspace, "kubelet")), </ins> fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "e2e_node.test")), fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "ginkgo")), fmt.Sprintf("/usr/bin/chcon -u system_u -r object_r -t bin_t %s", filepath.Join(workspace, "mounter")),

Yesterday Not yet available. Jobs fail to start due to kubelet startup issues. /sig
/sig testing node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We identified as a potential fix. These labels are what we actually want. But the tests actually started to pass before this merge. It still isn't clear to us what happenend to make these tests fail. and I were able to reproduce the test failures on FCOS but it seems to have resolved itself. Writing this here for record but it was an odd one thats for sure!


Question: // ScaleNamespaceListerExpansion allows custom methods to be added to // ScaleNamespaceLister. type ScaleNamespaceListerExpansion interface{} <del> // StatefulSetListerExpansion allows custom methods to be added to // StatefulSetLister. type StatefulSetListerExpansion interface{} // StatefulSetNamespaceListerExpansion allows custom methods to be added to // StatefulSetNamespaceLister. type StatefulSetNamespaceListerExpansion interface{} </del>

As a prerequisite of moving the , , , and controllers to use the apps/v1beta2 APIs, we need to add list expansions for , , , and to .
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps


Question: request, retry.DoExponentialBackoffRetry(&sendBackoff), ) <ins> if response == nil && err == nil { return response, retry.NewError(false, fmt.Errorf("Empty response and no HTTP code")) } </ins> return response, retry.GetError(response, err) }

What happened: We're seeing legacy-cloud-providers/azure/clients (in our case, the synchronised copy used by latest cluster-autoscaler (1.19)) segfaulting under heavy pressure and ARM throttling: In that specific case the cluster-autoscaler triggered the issue, but it can likely affect other ARM clients using that code, including in-tree kubelet or controller-manager. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.8 Cloud provider or hardware configuration: Azure OS (e.g: ): Ubuntu 20.04.1 LTS Kernel (e.g. ): 5.4.0-1022-azure -Ubuntu SMP Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/milestone v1.20


Question: registry = "k8s.gcr.io/build-image", repository = "debian-iptables", # Ensure the digests above are updated to match a new tag <del> tag = "buster-v1.3.0",  # ignored, but kept here for documentation </del> <ins> tag = "buster-v1.4.0",  # ignored, but kept here for documentation </ins> ) def etcd_tarballs():

What happened: The latest release of the debian-base image, v2.1.3, contains a vulnerable version of the apt package. CVE-2020- impacts apt version 1.8.2.1. What you expected to happen: Image should be updated to use at least apt 1.8.2.2 to clear the vulnerability. How to reproduce it (as minimally and precisely as possible):
/sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kube-proxy is using debain as base image. /assign Let me check other components
checked the base image in and I seems that no new debian or ubuntu official image is built for this CVE ?
That is correct, the purpose of this issue is to get a new image built and released, one which contains a resolution for the CVE.
per comments of @ tianon in Will update it once debain releases it.


Question: ginkgo.It(t.name, func(ctx context.Context) { m.init(ctx, testParameters{ registerDriver: test.deployClusterRegistrar, <del> podInfo:        test.podInfoOnMount}) </del> <ins> podInfo:        &test.podInfoOnMount}) </ins> ginkgo.DeferCleanup(m.cleanup) <del> waitUntilPodInfoInLog(ctx, m, test.expectPodInfo, test.expectEphemeral, 1) </del> <ins> waitUntilPodInfoInLog(ctx, m, test.expectPodInfo, test.expectEphemeral) </ins> }) } }) ginkgo.Context("CSI PodInfoOnMount Update", func() { <del> var ( podInfoTrue  = true podInfoFalse = false ) </del> tests := []struct { name              string <del> oldPodInfoOnMount *bool newPodInfoOnMount *bool </del> <ins> oldPodInfoOnMount bool newPodInfoOnMount bool </ins> }{ { name:              "should not be passed when update from true to false", <del> oldPodInfoOnMount: &podInfoTrue, newPodInfoOnMount: &podInfoFalse, }, { name:              "should not be passed when update from true to nil", oldPodInfoOnMount: &podInfoTrue, newPodInfoOnMount: nil, </del> <ins> oldPodInfoOnMount: true, newPodInfoOnMount: false, </ins> }, { name:              "should be passed when update from false to true", <del> oldPodInfoOnMount: &podInfoFalse, newPodInfoOnMount: &podInfoTrue, }, { name:              "should be passed when update from nil to true", oldPodInfoOnMount: nil, newPodInfoOnMount: &podInfoTrue, </del> <ins> oldPodInfoOnMount: false, newPodInfoOnMount: true, </ins> }, } for _, t := range tests {

1 ci-kubernetes-e2e-gci-gce 1 ci-cos-containerd-e2e-cos-gce in the master blocking test grid 1 ci-kubernetes-e2e-gci-gce-network-proxy-grpc 1 ci-kubernetes-e2e-gci-gce-proto Kubernetes e2e suite.[It] [sig-storage] CSI Mock workload info CSI PodInfoOnMount Update should be passed when update from false to true 12/14 code thaw after 1.29.0 release was merged at the time and is related to PodInfoOnMount. /cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
testCase: podInfoOnMount should be passed when update from false to true kubelet log: csi-mockplugin-0-mock log: the function in the test case is not working as expected. expect log at Line 2, but actual log is at Line 3.
is called twice for the same volume sometime even though the volume is already published successfully. Not only this issue, but also other issues like I wonder if it is a potential bug or a normal behavior. cc what do you think?
good catch. I think that any test that depends on exact index of NodePublish call will be flaky, sometimes Kubelet has to retry NodePublish. It would be better to check for NodePublish parameter changes. For example, test should check there were 1 or more NodePublish calls with no pod info and then 1 or more NodePublish calls with pod info afterwards.


Question: : ${KUBE_GCE_INSTANCE_PREFIX:="e2e-autoscaling"} : ${PROJECT:="k8s-jnks-e2e-gce-autoscaling"} : ${FAIL_ON_GCP_RESOURCE_LEAK:="true"} <ins> : ${ENABLE_DEPLOYMENTS:=true} </ins> # Override GCE default for cluster size autoscaling purposes. ENABLE_CLUSTER_MONITORING="googleinfluxdb" ADMISSION_CONTROL="NamespaceLifecycle,InitialResources,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota"

We need to write e2e tests which combines horizontal pod autoscaling with deployments (scale sub-resourcer should point to a deployment object). The tests should be similar to The library from should be extended and allow creation of a resource consumer as a deployment (currently it is always replication controller). The new tests should create a deployment resource consumer and pass a scale-sub-resource to the deployment to autoscaler.
part of
/cc
/cc FYI requests have been made to verify behavior at scale on *this.
Yeah that sounds like a good test to run re: scale out to n node cc also


Question: } } podStatus.Conditions = append(podStatus.Conditions, getPodReadyCondition(spec, podStatus.ContainerStatuses)...) hostIP, err := kl.GetHostIP() if err != nil { glog.Errorf("Cannot get host IP: %v", err)

Currently kubelet reports current status only. If pod's restart policy is not never, the last terminated container status will be override by the latest status, and lost. We should report last terminated container status. I am going to update our v1beta3 api first, but will hold the implementation until we have
did you have a chance to start working on it? IIUC, to support this we need to change the api (), which IIUC from what wrote in some other thread should be done this week. I'm happy to help with it if you want me to help.
Only breaking changes have a deadline. Additive changes can be done at any time.
I am working on this. PR should be ready today or tomorrow.
Also agree with This is not part of API breakage.
Thanks for answers & clarifications and


Question: "github.com/spf13/pflag" "k8s.io/kubernetes/contrib/mesos/pkg/executor/service" "k8s.io/kubernetes/contrib/mesos/pkg/hyperkube" <del> "k8s.io/kubernetes/pkg/util" </del> "k8s.io/kubernetes/pkg/util/flag" "k8s.io/kubernetes/pkg/util/logs" "k8s.io/kubernetes/pkg/version/verflag"

When I tried to use the following command to build kubernetes: I encountered the following errors: I'm using Ubuntu 14.04 and the code from master branch. will submit a patch then to fix this problem.
whoa, I just landed in the exact same place, I am trying to build k8s on a CentOS7 box. Tried with removing the unused utils and works like a charm !


Question: // in, ensuring the appropriate nested handler method is invoked. An object // that starts passing the filter after an update is considered an add, and an // object that stops passing the filter after an update is considered a delete. <ins> // Like the handlers, the filter MUST NOT modify the objects it is given. </ins> type FilteringResourceEventHandler struct { FilterFunc func(obj interface{}) bool Handler    ResourceEventHandler

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I discovered colleagues writing informer clients that modify objects delivered by informers. I went looking for where it says they should not do that, and I could not find it. What you expected to happen: I expected this critical restriction to be prominently documented. How to reproduce it (as minimally and precisely as possible): Try to find where it says that a client of an informer must not modify an object delivered by the informer. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Documentation aside, I would suggest adding to your CI infra to make it harder for people to make this mistake.
/assign Thank you for fixing docs!


Question: return } if status, ok := obj.(*metav1.Status); ok { <del> ae.ResponseStatus = status </del> <ins> // selectively copy the bounded fields. ae.ResponseStatus = &metav1.Status{ Status: status.Status, Reason: status.Reason, Code:   status.Code, } </ins> } if ae.Level.Less(auditinternal.LevelRequestResponse) {

We the into the field of audit events. This was done to capture the response status even when the response is omitted, but results in (potentially large) duplicate data being sent. In some examples, the status message is repeated in the status details, resulting in a long message being duplicated 4 times. I suggest that rather than copying the whole ResponseStatus, we selectively copy the bounded fields, specifically: status reason code Note that this is a breaking change, and if there is agreement should be done before graduating audit to stable. /kind bug /sig auth /priority important-soon
/cc
Could you go ahead and add "approved-for-milestone" label to this, as well as status? That will help it stay in the milestone if this is a 1.10 blocker. Thanks!
Moving it to 1.11, together with graduating audit logging API to stable
/sig auth /kind bug /milestone v1.11 /priority important-soon /status approved-for-milestone
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
No one is working on this? Mind if I assign it?
status check? This has maybe gone dormant. Is it still targeting 1.11?
Yes.


Question: if _, err := io.Copy(w, r); err != nil { return fmt.Errorf("failed to compress %q to %q: %v", log, tmpLog, err) } <ins> // The archive needs to be closed before renaming, otherwise an error will occur on Windows. w.Close() f.Close() </ins> compressedLog := log + compressSuffix if err := c.osInterface.Rename(tmpLog, compressedLog); err != nil { return fmt.Errorf("failed to rename %q to %q: %v", tmpLog, compressedLog, err) } // Remove old log file. <ins> r.Close() </ins> if err := c.osInterface.Remove(log); err != nil { return fmt.Errorf("failed to remove log %q after compress: %v", log, err) }

Currently, when kubelet will try to compress the logs to a .gz file, it will attempt to rename the archive before closing its file handles, which results in an error on Windows. An error should not occur while compressing the logs. This error can easily be reproduced by running the unit tests on Windows: /sig windows Kubernetes v1.22. N/A Any Windows OS. <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } zone := cloudprovider.Zone{ <del> FailureDomain: failureDomain, Region:        to.String(vm.Location), </del> <ins> FailureDomain: strings.ToLower(failureDomain), Region:        strings.ToLower(to.String(vm.Location)), </ins> } return zone, nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider gets the availability zone from IMDS directly and the zone is usually in lower cases in most regions. But there're chances this is not true in some regions, hence customer's workloads depend on availability zone may not work. What you expected to happen: The availability zone is always in lower cases. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/assign
Fixed by .


Question: "fmt" "math" "reflect" <ins> "sync" </ins> "testing" <del> "k8s.io/utils/pointer" </del> <ins> "github.com/google/go-cmp/cmp" </ins> dto "github.com/prometheus/client_model/go" <ins> "k8s.io/component-base/metrics" "k8s.io/component-base/metrics/legacyregistry" "k8s.io/utils/pointer" </ins> ) func samples2Histogram(samples []float64, upperBounds []float64) Histogram {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : run got the below error message, We should provide the metric data for different extension points from the metric of "schedulerframeworkextensionpointdurationseconds" Why is this needed: Those metrics have been removed by this commit - Those metrics are deprecated cause it is suggested to use - , but simply remove those metrics definitely is not a nice way to address the issue. /sig scheduling
/triage accept
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: path, err := ioutil.TempDir("", "FileStore") assert.NoError(t, err) defer cleanUpTestPath(t, path) <del> store := &FileStore{path: path} </del> <ins> store, err := NewFileStore(path) assert.NoError(t, err) </ins> Checkpoints := []struct { key       string

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: Using to start a cluster with only one node and found that there are some error logs in kubelet as following: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc
/cc
Seems caused by my env due to I cannot access gcr due to the proxy failed, will continue to check...
We should create the checkpoint directory when creating the checkpoint store, or else the list failed error will be quite spammy before a container is created.
Will push a fix soon.


Question: } } <ins> func getDefaultTestProbes(protocol, path string) []network.Probe { expectedProbes := []network.Probe{ { Name: to.StringPtr("atest1-TCP-80"), ProbePropertiesFormat: &network.ProbePropertiesFormat{ Protocol:          network.ProbeProtocol(protocol), Port:              to.Int32Ptr(10080), IntervalInSeconds: to.Int32Ptr(5), NumberOfProbes:    to.Int32Ptr(2), }, }, } if path != "" { expectedProbes[0].RequestPath = to.StringPtr(path) } return expectedProbes } func getDefaultTestRules(enableTCPReset bool) []network.LoadBalancingRule { expectedRules := []network.LoadBalancingRule{ { Name: to.StringPtr("atest1-TCP-80"), LoadBalancingRulePropertiesFormat: &network.LoadBalancingRulePropertiesFormat{ Protocol: network.TransportProtocol("Tcp"), FrontendIPConfiguration: &network.SubResource{ ID: to.StringPtr("frontendIPConfigID"), }, BackendAddressPool: &network.SubResource{ ID: to.StringPtr("backendPoolID"), }, LoadDistribution:     "Default", FrontendPort:         to.Int32Ptr(80), BackendPort:          to.Int32Ptr(80), EnableFloatingIP:     to.BoolPtr(true), DisableOutboundSnat:  to.BoolPtr(false), IdleTimeoutInMinutes: to.Int32Ptr(0), Probe: &network.SubResource{ ID: to.StringPtr("/subscriptions/subscription/resourceGroups/rg/providers/" + "Microsoft.Network/loadBalancers/lbname/probes/atest1-TCP-80"), }, }, }, } if enableTCPReset { expectedRules[0].EnableTCPReset = to.BoolPtr(true) } return expectedRules } </ins> func getTestLoadBalancer(name, rgName, clusterName, identifier *string, service v1.Service, lbSku string) network.LoadBalancer { lb := network.LoadBalancer{ Name: name,

What would you like to be : Support for additional load balancer health probe configurations (specifically HTTP and HTTPS) for Azure Load Balancers. Why is this needed: The generated Azure Load Balancers for type services only support TCP health probes. Those guarantee that the backend pool consists of nodes listening on the target port but do not validate whether or not anything is responding. We've also noticed elevated volumes SSL handshake errors generated by the TCP health checks targeting 443 when using more verbose log level configurations in ingress-nginx. It would be valuable to configure the type of health probe used by the AKS-created Azure Load Balancer.
/sig azure
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Is there any update on this?
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
what do you think about this feature requests?
I had a look at a possible implementation. I did not create a PR yet, because the relevant files are in and the said there should only be bug fixes, no new features. In , there is some logic which would lead to the load balancer created with protocol HTTP (which is what we want) The check is from here: and the "RequestsOnlyLocalTraffic" is about the : However, I do not understand the relationship here, why setting should result in HTTP health check. What I think we could do is add a new annotation like and when this is set to , the health check will use the HTTP protocol in .

We could also add an additional annotation to set the property of the health check probe, this could be used so that people can set the path to or similar.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
We have a similar requirement for having an http health probe with a custom path. According to comment, we could get an http probe by setting but our service expects health probes on a different path, not . Looks like proposed solution would work for us too. Are there any workarounds to achieve this currently?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign


Question: defer createWait.Done() var err error <del> podTemplate := &template </del> <ins> podTemplate := template.DeepCopy() </ins> if utilfeature.DefaultFeatureGate.Enabled(features.ScheduleDaemonSetPods) { <del> podTemplate = template.DeepCopy() </del> // The pod's NodeAffinity will be updated to make sure the Pod is bound // to the target node by default scheduler. It is safe to do so because there // should be no conflicting node affinity with the target node.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Create a DaemonSet. It'll create pods with schedulerName=default-scheduler. However, the DaemonSet pods aren't actually scheduled by default-scheduler but DaemonSet controller.
If the "schedulerName" is empty, apiserver will set the default value to "default-scheduler". To avoid the confusion, I think we can let DaemonSet set this field (for example 'ds-controller') when creating the DaemonSet pods, WDYT?
What's "schdulerName" for? Is it just for telling the users who scheduled this pod, or is this information used by other resources in the system? Also, I'd expect this field to be set by schedulers. Why is it not set by default scheduler, but apiserver? How does apiserver know which scheduler does the work, if there are more than one? Letting DaemonSet controller set this field sounds reasonable, as DaemonSet controller is the "scheduler".
is introduced by multi-scheduler; a common case is to set it by user in : it means this Pod should be scheduled by specific scheduler. In each scheduler (including the default scheduler), it will check whether it's responsible for the pending Pod. The default value is default scheduler :) Sure, I'll open a PR for this :)
/assign
Isn't scheduler name ignored when spec.nodeName is set?
no, right now; that makes sense, I'll also include this in PR :)
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity.


Question: gotFitErr, gotOK := err.(*framework.FitError) wantFitErr, wantOK := test.wErr.(*framework.FitError) if gotOK != wantOK { <del> t.Errorf("Expected err to be FitError: %v, but got %v", wantOK, gotOK) </del> <ins> t.Errorf("Expected err to be FitError: %v, but got %v (error: %v)", wantOK, gotOK, err) </ins> } else if gotOK { if diff := cmp.Diff(gotFitErr, wantFitErr); diff != "" { t.Errorf("Unexpected fitErr: (-want, +got): %s", diff)

When Extender filters out some Nodes, we don't set any unschedulable plugins at all. It means Extender is completely ignored during the requeueing process. So, what's happening is: If Extender filters out all Nodes during scheduling, this Pod is soon retried because this Pod doesn't have any plugin name in unschedulable plugins. If Extender filters out some Nodes during scheduling and plugins filter out all other Nodes, this Pod is retried based on plugins' QueueingHint. Even if any cluster events happen and it could change Extender's decision (but not any plugins' decision), this Pod won't be requeued to activeQ/backoffQ. The latter case is serious because it could make Pods being stuck in unschedulable pod pool in 5min. We should have a short-term solution for the latter case. We can requeue Pods, which were rejected by Extender, by any kind of cluster events because we cannot know which events make Pods schedulable. Eventually, this issue makes us wonder how Pods rejected by Extender should be requeued. We cannot implement QHint equivalent in Extenders because it'd be too slow to call Extender every time any cluster events happen. Probably, somehow implementing EventsToRegister equivalent in Extender? Use Extender which does something in Filter. No response master <detailsn/a </details<details</details<details</details<details</details<details</details
/assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/reopen We'll close it when all cherry-picks are done.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close Cherry-picks are done.


Question: nonMatchingUsername := "foo" ginkgo.By("creating a testing prioritylevel") <del> createdPriorityLevel, err := f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Create( context.TODO(), &flowcontrol.PriorityLevelConfiguration{ ObjectMeta: metav1.ObjectMeta{ Name: testingPriorityLevelName, }, Spec: flowcontrol.PriorityLevelConfigurationSpec{ Type: flowcontrol.PriorityLevelEnablementLimited, Limited: &flowcontrol.LimitedPriorityLevelConfiguration{ AssuredConcurrencyShares: 1, // will have at minimum 1 concurrency share LimitResponse: flowcontrol.LimitResponse{ Type: flowcontrol.LimitResponseTypeReject, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) defer func() { // clean-ups err := f.ClientSet.FlowcontrolV1beta1().PriorityLevelConfigurations().Delete(context.TODO(), testingPriorityLevelName, metav1.DeleteOptions{}) framework.ExpectNoError(err) err = f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Delete(context.TODO(), testingFlowSchemaName, metav1.DeleteOptions{}) framework.ExpectNoError(err) }() </del> <ins> createdPriorityLevel, cleanup := createPriorityLevel(f, testingPriorityLevelName, 1) defer cleanup() </ins> ginkgo.By("creating a testing flowschema") <del> createdFlowSchema, err := f.ClientSet.FlowcontrolV1beta1().FlowSchemas().Create( context.TODO(), &flowcontrol.FlowSchema{ ObjectMeta: metav1.ObjectMeta{ Name: testingFlowSchemaName, }, Spec: flowcontrol.FlowSchemaSpec{ MatchingPrecedence: 1000, // a rather higher precedence to ensure it make effect PriorityLevelConfiguration: flowcontrol.PriorityLevelConfigurationReference{ Name: testingPriorityLevelName, }, DistinguisherMethod: &flowcontrol.FlowDistinguisherMethod{ Type: flowcontrol.FlowDistinguisherMethodByUserType, }, Rules: []flowcontrol.PolicyRulesWithSubjects{ { Subjects: []flowcontrol.Subject{ { Kind: flowcontrol.SubjectKindUser, User: &flowcontrol.UserSubject{ Name: matchingUsername, }, }, }, NonResourceRules: []flowcontrol.NonResourcePolicyRule{ { Verbs:           []string{flowcontrol.VerbAll}, NonResourceURLs: []string{flowcontrol.NonResourceAll}, }, }, }, }, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err) ginkgo.By("response headers should contain flow-schema/priority-level uid") </del> <ins> createdFlowSchema, cleanup := createFlowSchema(f, testingFlowSchemaName, 1000, testingPriorityLevelName, matchingUsername) defer cleanup() ginkgo.By("checking response headers contain flow-schema/priority-level uid") </ins> if !testResponseHeaderMatches(f, matchingUsername, string(createdPriorityLevel.UID), string(createdFlowSchema.UID)) { framework.Failf("matching user doesnt received UID for the testing priority-level and flow-schema") }

Should we create tests around the creation, modification and deletion of the objects ? For example, adding an invalid FlowSchema with a misconfigured PriorityLevelConfiguration and verifying that the Status appropriately reflects its invalidity. I’m assuming the controller tests will suffice and we don’t need explicit e2e tests for this. What about CRUD tests wherein we’re validating the debug endpoints are showing the right values ? The first class of tests are the basic ones to verify that the requests are being categorized into the right FlowSchema and Priority level. We’ll set up a new PriorityLevelConfiguration and FlowSchema objects and observe that requests are correctly being classified onto them. Verification that requests are correctly being classified will be tricky. For now, we can scrape the metrics endpoint for ‘’ metric to verify that the expected number of requests for a test FlowSchema is correct. Another option is to utilize the ‘dump_requests’ debug endpoint to verify the correct request classification. If during the course of writing the tests we need additional information, we can extend the debug endpoint. Scraping the metrics endpoints to validate the proper request categorization could lead to a fickle e2e test. So querying the ‘debug’ endpoint seems like the right way to go. This test will also serve as a proxy for an end-to-end test that ensures that requests matching a new FlowSchema are successful. Does this have the propensity to be a flaky test(s) ? This class of tests will focus on validating the behavior this feature promises, viz., requests of low priority won’t be drowned out by requests of higher priority and a single bad actor won’t detrimentally affect the requests from others. The following tests will ensure that high velocity flows don’t affect the low velocity flows. For this test, we’ll have two entities submitting requests at the same time. These requests will map to two different FlowSchemas at different priority levels. The higher priority flow will have a lot of requests going through it, far above its concurrency share. At the same time the low priority flow will be submitting requests as well at a rate that well within the concurrency limits of that Flow. The test will ensure that the low velocity (low priority) flow succeeds. This test will be the same as the above one except that the two flows will map to the same FlowSchema but are separated via their Flow Distinguisher.

One of them will be a high velocity flow which the other is a low velocity one. The test will ensure that the low velocity requests still get through and succeed. Does this have the propensity to be a flaky tests ? If the test fails, will the runner of this test be able to grok where the problem could lie and debug ? Is this an actionable failure ?
cc/
We don't need to test that invalid items are rejected, unit tests are better for that. We should try to exercise the basic features (exempt, queue...) We should not rely on the debug endpoint, since we want this to be a conformance test eventually Metrics scraping is also a little questionable. I think we should exercise fair queueing (make two new PLs at different shares, saturate both, verify the responses follow the share ratio). I think we should exercise the shuffle sharding (make a new PL, saturate it with one/few elephants and many mice, verify the mice get service. (those latter two seem to match your last two.) This would be a pretty high level test that tests things end users will care about, but I think it should still be quite hard to pass without actually running our code.
/sig api-machinery /area test /kind feature
cc
to verify the requests can be correctly classified, how about we make use of the ? we can do a test like: make two PLs, and two clients w/ different identities then make the clients are hitting different FS/PL.
what about using a test aggregated apiserver to serve different PLs, and with different the response latency
No strong opinion on this one. I am not sure if that was a response to the ideas in the "Basic CRUD" section --- there the idea is not about testing validation but rather testing the status setting about referential integrity. The debug endpoints are about what is happening at the moment of the GET. Controlling that tightly enough to be worth a test would be tricky. The only plausible approach in an E2E test would be to register an authorization webhook that synchronizes with the test function. I like the basic idea. For the reasons above, I would not recommend looking at the debug endpoints. I do not understand what is wrong with scraping the metrics. Are they not part of conformance? This feature does add response headers.

Are those part of conformance? I like the basic idea. First, remember that this feature does not control velocity; it controls the product of velocity and execution duration. To get a high signal-to-noise ratio, we would want deterministic and large execution durations. Looking in , you can see that authorization is included in execution. So we could add a controlled large amount of duration in there. I am a little surprised at the question of "If the test fails, will the runner of this test be able to grok where the problem could lie and debug"; why would that be more of a question here than for any other E2E test? Presumably the test failure message will say something about the expected and observed behavior. I think the drown-out scenario involves testing queuing. I think exempt was not covered above, and would be a worthwhile addition. This mixes up two things. Fair queuing is something that happens within one priority level. This sounds like it may be expensive for an E2E test. I think we can make a much more efficient unit test of QueueSet that exercises shuffle sharding. Because the QueueSet implementation can be given a FakeEventClock, we can run the test much faster than real time. To get a high signal-to-noise ratio we want controlled, high latencies. Making an aggregated apiserver do that is possible. Perhaps the simplest approach would be to put a fixed sleep in the registry strategies for whatever custom resources the aggregated apiserver is serving. But this seems like a lot more work than necessary. I think an authorization webhook is a simpler approach.
could you elaborate a bit more about the authorization webhook approach? it is only called in the authorization path, if i understand it correctly.
Authorization, and admission control, happen after the priority and fairness filter.
is this really possible? I am likely missing something, but my understanding from the is that each level is limited to the concurrency shares assigned to it.
: the text should be revised to more clearly express the intent. It is that the driving client has many more requests outstanding at once than the corresponding priority level will allow to be served concurrently.
FYI: I'm working on the second test.


Question: <ins> apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: example spec: backend: serviceName: default-backend servicePort: 80 rules: - http: paths: - path: /testpath backend: serviceName: test servicePort: 80 </ins>  No newline at end of file

What happened: Tried to convert a v1beta1 Ingress object to v1, and received the following error: What you expected to happen: The conversion to work. How to reproduce it (as minimally and precisely as possible): Then run: /sig cli /assign
Can i work on this issue?
/assign
/triage accepted


Question: "//pkg/api/v1:go_default_library", "//pkg/apis/extensions/v1beta1:go_default_library", "//pkg/client/clientset_generated/clientset:go_default_library", <ins> "//pkg/cloudprovider:go_default_library", </ins> "//test/e2e/chaosmonkey:go_default_library", "//test/e2e/common:go_default_library", "//test/e2e/framework:go_default_library",

In federation tests, if we observe the build logs. we see cleaning up LoadBalancer from cloud provider seems not working. It has 2 effects: delays the tests by waiting to delete but always returns error. so we waste like 5 minutes. maybe we are leaking the LoadBalancer in the project.
we can see these errors in every build. but just for reference here are some build logs https://k8s-
cc
All the errors in logs are . It doesn't necessarily indicate we are leaking resource, just now we must have clusterID initialized before calling EnsureLoadBalancerDeleted(). Would be better to not call this function directly in e2e test :)
Federation cleanup should use the same implementation as non-federated e2e test:
function looks like only specific to GCE. in future we would run the e2e tests in AWS too. it would be better to use generic functions if any.
using is working for us. here is the test logs https://k8s-
EnsureLoadBalancerResourcesDeleted() actually cleans up nothing, it only lists forwarding rules and probably logging :)


Question: events, _ = dd.client.CoreV1().Events(namespace).Search(scheme.Scheme, d) } <del> return describeDeployment(d, selector, d, events, dd) </del> <ins> return describeDeployment(d, selector, d.DeepCopy(), events, dd) </ins> } func describeDeployment(d *appsv1.Deployment, selector labels.Selector, internalDeployment *appsv1.Deployment, events *corev1.EventList, dd *DeploymentDescriber) (string, error) {

Create a deployment as described in section, and run 'kubectl describe deploy/testWhat happened: shows , while shows the current replicaset infomation What you expected to happen: shows the current replicaset infomation, while shows How to reproduce it (as minimally and precisely as possible): Create a deployment as follows, and wait its pod started. Then, run 'kubectl describe deploy/test Anything else we need to know?: Environment: Kubernetes version (use ): kubectl of 1.14+(I did not try older versions) Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli
/assign
Ok, can confirm here. With the old code, during a rollout the data is shown at OldReplicaSets (both the old and new RS) while with the patch during the rollout the NewReplicaSet contains the net one and the Old the old one ( obvious here) while the old exists. I'll make some more tests here and check if is there anything else in your PR that needs better verification, otherwise wer can follow with it! Thank you!
/triage accepted


Question: // alter the kubelet state at all. func (kl *Kubelet) convertStatusToAPIStatus(pod *v1.Pod, podStatus *kubecontainer.PodStatus) *v1.PodStatus { var apiPodStatus v1.PodStatus <del> apiPodStatus.PodIPs = make([]v1.PodIP, 0, len(podStatus.IPs)) </del> <ins> // The runtime pod status may have an arbitrary number of IPs, in an arbitrary // order. Pick out the first returned IP of the same IP family as the node IP // first, followed by the first IP of the opposite IP family (if any). podIPs := make([]v1.PodIP, 0, len(podStatus.IPs)) var validPrimaryIP, validSecondaryIP func(ip string) bool if len(kl.nodeIPs) == 0 || utilnet.IsIPv4(kl.nodeIPs[0]) { validPrimaryIP = utilnet.IsIPv4String validSecondaryIP = utilnet.IsIPv6String } else { validPrimaryIP = utilnet.IsIPv6String validSecondaryIP = utilnet.IsIPv4String } </ins> for _, ip := range podStatus.IPs { <del> apiPodStatus.PodIPs = append(apiPodStatus.PodIPs, v1.PodIP{ IP: ip, }) </del> <ins> if validPrimaryIP(ip) { podIPs = append(podIPs, v1.PodIP{IP: ip}) break } </ins> } <del> if len(apiPodStatus.PodIPs) > 0 { apiPodStatus.PodIP = apiPodStatus.PodIPs[0].IP </del> <ins> for _, ip := range podStatus.IPs { if validSecondaryIP(ip) { podIPs = append(podIPs, v1.PodIP{IP: ip}) break } </ins> } <ins> apiPodStatus.PodIPs = podIPs if len(podIPs) > 0 { apiPodStatus.PodIP = podIPs[0].IP } </ins> // set status for Pods created on versions of kube older than 1.6 apiPodStatus.QOSClass = v1qos.GetPodQOS(pod)

What happened: The is taken from without regard for the main family of the cluster. No warning or error is given if gets the wrong family. What you expected to happen: That is taken from the "main" ip family, or at the very least the POD is not started and an error is given if it's not. How to reproduce it (as minimally and precisely as possible): Use the CNI-plugin and ipam. Use a single node cluster to avoid that assign the same address to PODs in different nodes. This may be CRI-plugin dependent, I use . The PODs are assigned dual addresses which has been supported since v1.9. Note the order; IPv6 first. In a dual-stack or a single-stack cluster with main family IPv4 will get an IPv6 address even if the main family is IPv4; Anything else we need to know?: First, note that this is not a dual-stack problem. may get the wrong family in a single-stack cluster if the PODs has dual addresses. The bug may however have been introduced with the dual-stack support, before K8s may have selected the correct family, I have not checked that. This is reported and discussed in There it is considered as a CNI-plugin problem, the CNI-plugin must present addresses in the order K8s wants them. Problem is that CNI-plugins doesn't know this. The addresses takes the path; The problem can be addressed in either of these places. The current situation is that the full responsibility lies on the , it must send the addresses in the "correct order". This is not the best place. The CNI-plugins are not a K8s-only thing and to require that a "prefered-family" or something similar must be supported by CNI-plugins to be "Kubernetes compliant" should be avoided. The CRI-plugin is a K8s-only thing and a "prefered-family" can be introduced so the CRI-plugin can sort the address array before sending it to . This is however undesirable since it adds a configuration complexity, it put a responsibility on the user or installation tool to configure the "prefered-family" to all CNI-plugins now and in the future. The best is to handle this in K8s itself where the "main" family is known. Is this really necessary?

All communication works even if is not of the main family. EndpointSlices seem to get this right regardless of order. Endpoint on the other hand seem to take the (old) podIP. So perhaps; is sufficient? If really must be , then K8s should sort the array so belongs to the main family. The problem exist on all supported K8s versions and on "master". If installed in the default way the situation is; Calico always send IPv4 first Cilum always send IPv6 first I don't think you can control the order, but I have not asked. Environment: Kubernetes versions: v1.17.12, v1.18.9, v1.19.2, v1.20.0-alpha.1, master v1.20.0-alpha.1.257+ Cloud provider or hardware configuration: None OS (e.g: ): xcluster Kernel: linux-5.8.1 Install tools: None Network plugin and version: bridge, host-local v0.8.7 Others: CRI-plugin: cri-o 1.18.3
/sig network
For the record; all communication via services works perfectly fine even if the order in is "faulty" with not belonging to the main family in k8s v1.18.8.
I think this is a duplicate of CNI plugins must report multiple IPs in the order expected by the cluster. The podIPs[0] == podIP requirement is for backwards compatibility with clients that do not read podIPs. Server-side reordering causes problems with clients that continually write/reconcile until the server list exactly matches their local list.
The problem exist in single-stack K8s clusters when PODs has dual addresses (supported since K8s v1.9, i.e way before K8s dual-stack which came in v1.16) And this issue is not a duplicate of . That issue is about a problem when the CNI-plugin uses the "correct" order. This issue is about the fact that K8s puts undocumented requirements on the CNI-plugins. There is no requirement on order in the . And since this requirement is undocumented it is not supported for instance by cilium and calico. The itself should be the requirement is for backwards compatibility with clients that do not read podIPs.

Since they doesn't read podIPs the value of should not matter to them, or? I think this should be handled in K8s, but if the responsibility is pushed downhill to the CNI-plugins then two things must be done in K8s; extra requirement on "Kubernetes compliant" CNI-plugins (outside the CNI spec) must be formally documented and communicated to the CNI-plugin providers. a "faulty" CNI-plugin accidently put's an IP-address of the "wrong" family in the first item then K8s must report an error and not start the POD. Please note again that this is not a K8s dual-stack problem. In a single-stack IPv6 K8s cluster the CNI-plugins must provide an IPv6 address first in the list.
The reason podIP must match podIPs[0] is related to updates to an existing object by an old client. See for a related problem when this was not done. An old client updated an existing object, setting only the podIP field (which is the only one it was aware of). The API server needed to detect that and populate the podIPs field correctly for new clients to read. The detection is based on the mismatch between podIP and podIPs[0], and the server responds by populating . That means that new clients (aware of both fields) must set podIP and podIPs[0] to match.
So this is weird, but... dual-stack clusters do not actually have a "main family", because decisions about IP families are made by multiple components, and kubernetes clusters do not have a single source of configuration truth that applies to all components. So it is explicitly allowed to have kube-apiserver default to IPv4-primary IPv6-secondary for services, and have your network plugin default to IPv6-primary IPv4-secondary for pod IPs, and to have some nodes have a single IPv4 address and some have a single IPv6 address and some have dual-stack addresses. It's not super clear why you'd want to do that, but we decided we had to allow it, simply because there's no easy way to prevent it. Of course in scenarios where the entire cluster is being installed/deployed by a single tool, one would expect that tool to configure all of the components in a consistent way. There is no "order K8s wants them".

Kubernetes doesn't want the addresses in any order. You want them in a particular order. Therefore you must configure the CNI plugin to return them in that order. Eg, in the example above, you should reverse the two in the config. Kubernetes's interpretation of this is that if Cilium sends the IPv6 address first, then it is telling Kubernetes that Kubernetes ought to prefer the IPv6 one. It appears that the CNI spec doesn't actually say that though... It doesn't seem to say anything about the ordering of multiple IPs... So maybe that should be clarified. If the CNI authors feel that the ordering of returned addresses is not meaningful then perhaps kubelet's behavior could be changed.
Thoughts on this?
CNI doesn't make any statement about ordering of IPs. It definitely doesn't have any notion of a "primary IP". It seems that containerd itself does have some
I mean that's literally what I just said :stuckouttongue:. The question is more if that is intentional and/or changeable. It seems strange that the decision about whether to allocate IPv4 and/or IPv6 IPs is left up to the network plugin, with no input from the runtime, but the decision about what to do with IPv4 and/or IPv6 IPs is then left up to the runtime, with no input from the network plugin.
That is very good an as it should be but... The quote is from the initial commant. Please note that this is a single-stack cluster! And that has a main family even though the PODs have dual addresses.
Right; don't comment on issues just before going to bed. Ooops. I don't think we want to introduce the notion of IP ordering / or preference in to CNI for a few reasons -- It's a bit of a breaking change, depending on how you read the spec It would be awkward to push the "preferred family" from k8s all the way down through to CNI It doesn't match how Linux works - interfaces don't have primary IPs (legacy aside). So fixing this in the apisever is probably the right place? BTW, what, exactly, defines a cluster's address family? ClusterCIDR, but that's optional (as is PodCIDR). ServiceCIDR, certainly. Is there an explicit config flag?

no flag, you configure the cluster with IPv4 or IPv6 addresses, of course if you have pod with IPv4 addresses and service with IPv6 that will not work kubelet set the status.PodIPs In a single-stack cluster you don't have a main family, you only have ONE family If we don't have PodIPs and only PodIP in a single-stack cluster this will be clearly a configuration problem in the CNI,no? I personally, think that this is a configuration problem and that we created it because we pluralised PodIP without a feature gate
Yes, and this is how it looks in an IPv4 single-stack cluster ( NOT set) if the CNI-plugin has been configured for dual addresses on PODs but returns the assigned addresses in the "wrong" order (ipv6 first);
Why is requesting an IPv6 address for pods in an IPv4 cluster not a configuration error? To me it seems a bit like running an ARM image on an AMD64 machine; Kubernetes will go through some motions but will not make it work or tell you what you did wrong.
Dual addresses in PODs has been supported from K8s v1.9, i.e. long before K8s dual-stack so that is not a configuration error. In the CNI specification there is nothing about required order for addresses or "prefered family" so it is not a configuration error for the CNI-plugin either. For K8s functions the order in does not matter. But design of upgrade requires that see Problem is that K8s (kubelet) sets "podIP = podIPs[0]" without checking the family. Here some error should be in place, like "CNI-plugin not compatible with K8s" and the POD should not be started, or better, sort the addresses or remove the requirement. This is a K8s problem. CNI-plugins should not have to care about order of reported addresses at least not without a CNI specification update.
This doesn't appear to answer my question. What is the meaning of an IPv6 pod address in an IPv4 cluster?
A cluster does not have an address family. Usually it seems like it does but that's just because the administrator has configured all of the individual components in a consistent way. But most components' configuration only affects what that component does (though in some cases that will have knock-on effects).

kube-controller-manager sets based on its (assuming is set) The network plugin (indirectly) sets (which may be affected by kcm's choice of ) based on its own configuration Either kubelet or an external cloud provider sets : based on , if passed, in the non-cloud-provider case based on cloud-provider-specific configuration (or hardcoded defaults) in the cloud case kubelet sets from kube-apiserver sets based on its kube-controller-manager creates endpoints, mostly based on the IP families chosen by kube-apiserver, except in the case of headless selectorless services, where it lets the user set endpoints of whatever family it wants, because it doesn't usually know whether the cluster is single or dual stack. kube-proxy determines whether rules are actually created for IPv4 and/or IPv6 services, based on the passed in if the dual stack feature gate is disabled, and unconditionally dual stack if the feature gate is enabled the cloud provider determines whether services get IPv4 and/or IPv6 IPs, based on cloud-provider-specific config or hardcoded defaults. (EDIT: well, and probably ) This... is kind of a mess, but there's no real way to fix it centrally given the way that kubernetes works. But we could add a config option to kubelet to unambiguously specify whether it should act like it's in a SSIPv4/SSIPv6/DSIPv4Primary/DSIPv6Primary cluster, and then make assignment make use of that.
Oof. Well. Maybe it makes sense for kubelet to stop setting PodIP, and leave that decision to the apiserver? It could make that decision based on ServiceCIDR, and order the families accordingly?
No, the dual stack KEP even explicitly calls out the fact that that it doesn't work that way: There is no single source of truth for kubernetes cluster config, and thus there is no single truth about whether the cluster is IPv4, IPv6, dual-stack-IPv4-primary, or dual-stack-IPv6-primary. If we want kubelet to impose some policy on PodIPs, then we need to tell it what policy to impose, because it has no entirely-reliable way of determining what policies any other component is imposing.
That is still much better than to require this function from the CNI-plugin which is not even K8s and follows a specification.

Please note that the networking e.g services, external access, works fine whatever order are in and whatever value has. I have not found any communication problem in K8s = v1.19 caused by this. The value of seem more "cosmetic". But it looks wrong to see ipv6 addresses of PODs in an ipv4-only cluster (). The problem seem to be an unfortunate upgrade design that imposed the constraint If that can some way be relaxed this becomes a no-problem as I see it.
/triage accepted /assign
What is the material impact of this behavior?
we can not relax because that will break existing clients. a side note. Using dualstack configured CNI on a none dual stack cluster was pre dualstack hack, if it must remain in use then CNI should only report one IP not two.
what's your reasoning to change CNI rather than kubelet?
We can't just put undocumented requirements on the CNI-plugins, they are not part of K8s, they have a formal specification and they have other users beside K8s. At least K8s must formally document the extra requirements on CNI-plugins that K8s has for the CNI-plugin to be K8s compliant.
As highlighted on the discussion there is no single truth of what clusterIP family is. Also kubelet will have to depend on which is not always used by CNIs. You can have a perfectly functional cluster that does not use PodCIDRs at all (I believe Cilium for example does not use it). Also I am not saying change CNI. A well configured CNI (single family for single stack cluster) will not have this problem.
Is the latter statement a retraction of the former? If not, how can they be reconciled?
He's saying that an administrator deploying a CNI plugin in a single-stack kubernetes cluster ought to configure that CNI plugin to return a single IP address, of the family being used by the kubernetes cluster. A CNI plugin that does not allow you configure what address families it uses may not be compatible with all kubernetes clusters. Eg, a CNI plugin that only supports IPv4 cannot be used as the network plugin in a single-stack IPv6 kubernetes cluster.
So, to sum up: Kubelet will always use the first IP returned by the CNI plugin as , regardless of any other cluster configuration.

Some CNI plugins, when configured to do dual stack, do not give you a way to configure which IP comes first, meaning kubelet may pick a that is not the IP family the admin considers the cluster's primary IP family In dual-stack clusters, this does not appear to cause any actual problems, other than that the "default" pod IPs shown in "" may not be the ones the administrator wants Nothing in the current kubelet configuration is supposed to indicate whether kubelet should prefer IPv4 or IPv6 pod IPs. It would be possible to add a new config option indicating this, or we could just declare that should also affect the sorting of pod IPs, because really why wouldn't you want that? In single-stack clusters, it has been suggested that the right answer is "don't do that then"; ie, the administrator should configure the CNI plugin to only return a single IP, and if they can't do that, then that's the CNI plugin's problem, not Kubernetes's
oh, and: if the CNI plugin returns multiple IPs which are not a dual-stack pair (eg, if it returns 2 IPv4 IPs, or 1 IPv4 IP and 2 IPv6 IPs), then kubelet will fail to update the pod at all, because it assumes it can just copy the IPs returned from CNI into , but the apiserver will only accept if it is either a single IP or a dual-stack pair.
That would be excellent! Then it would not be necessary to stop supporting dual-stack pods in a single stack K8s. And Cilium users wanting to upgrade their ipv4-only cluster to dual stack would not suddenly see ipv6 addresses on a .
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Just for the record, this is an actual problem when trying to deploy rook-ceph. Rook makes strong assumption about family being the same as family and just fails to create Ceph cluster when they do not match. are configured to listen only on but the advertised addresses are the corresponding service .

This result in listening only on pod IPv6 address but being advertised using the service IPv4 address.


Please write a test that: Starts kube-apiserver Adds a webhook which blocks everything it is possible to block. Re-starts kube-apiserver Verifies that kube-apiserver comes back up Verifies that it can remove the broken webhook. This would have caught
cc
Hi I'm new to kubernetes but eager to contribute. I think this will be a good first issue for me but I'll need some guide on how to proceed I will need the first two points explained and also which file I'm supposed to be making my changes to.
Here is a good example of how to start a realistic kube-apiserver in integration tests:
"Verifies that it can remove the broken webhook." sounds dangerous. The apiserver should come up and it should be possible to remove a webhook via the API. But removing a potentially broken (or intentionally DoS'ed) webhook asks for security attacks.
I interpreted "Verifies that it can remove the broken webhook" as just ensuring that no webhook can be registered which could potentially deny its own (or another webhook in the same namespace's) deletion. Broken webhooks definitely shouldn't be treated differently in terms of determining who can delete them, but validation should probably prevent webhooks from applying to creates, updates and deletes of webhook objects, wildcard selectors should exclude webhook objects, etc. Also, this behavior would need to be documented. I think the safest way to ensure that a webhook does not get deleted is with RBAC rules, rather than using a validating webhook to prevent its deletion. I can't think of a legitimate use case for webhooks which affect the admission of webhook objects, it seems like any workflow which relies on that could be implemented in another, easier to reason about way. thoughts?
I have the same understanding. Excluding webhooks from webhooks seems reasonable. But it's only one piece in the (restart when webhook broke) puzzle. We reconcile RBAC rules, create namespaces (or keep them when they exist already). So I expect some more exceptions.
Is this still something you are working on?
If you can make a webhook which prevents the apiserver from starting at all, this is a problem, no matter what the webhook is verifying/doing, because you can't recover from that state.

I'm not suggesting that the system should automatically remove a broken webhook, which would be a security problem, rather that it needs to be possible for an appropriately authenticated sysadmin to do so. Especially since making a broken webhook non-broken may require executing API calls that the broken webhook is preventing.
/assign
as you assigned this to you, can you sketch what the plan is for this?
The plan was to use an integration test, (i.e. ) which I implemented in but it turns out that since the integration test apiserver does not follow exactly the same startup chain as the real one, that it is insufficient to ensure the apiserver can actually come up with a broken webhook installed. The integration test doesn't even fail against the code before merged. So currently the plan is to do it as an e2e test, probably with the [Slow] and [Destructive] tags.
it should be 99.9% the same. What are you missing? I think an integration test would be much better than an e2e test. We have much more control there.
I think the bug fixed was something a controller (bootstrap controller?) was doing, not the apiserver
those bootstrap controller are also executed by the testserver. So I would expect that the bug could be hit in testserver integration tests too. Compare with the real start code and for the test server start code. They are the same. The only difference is that we tweak the options beforehand a bit to run with a random port and random etcd prefix.
Hmm, then I think the reason that the test doesn't fail pre- is because the post start hook changed by that PR isn't called unless api-aggregation is being used. I guess the integration test for this will have to do that as well. Although the one in is still somewhat valuable to make sure new startup namespace creations in non aggregated apiservers are done correctly.
- you probably already thought about it, but one more thing to ensure here is that: apiserver can starte responding "OK" to "/healthz" even if all webhooks are broken [that means e.g. finishing all post-startup hooks] Without that, apiserver would be consistently being killed + restarted by Kubelet on which it is running (at least in some setups).

Yeah, the post-start hooks are the thing that inspired this issue :) Wojciech Tyczynski <:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen : — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // being upgraded, then the container runtime must also support the initially // selected version or the redial is expected to fail, which requires a restart // of kubelet. <del> func (r *remoteImageService) determineAPIVersion(conn *grpc.ClientConn) error { </del> <ins> func (r *remoteImageService) determineAPIVersion(conn *grpc.ClientConn, endpoint string) error { </ins> ctx, cancel := getContextWithTimeout(r.timeout) defer cancel()

When installing the latest component, got the error: Then the flag , got error: It is confusing to see these error msgs at the first glance, need to make it more clearly. Add the required field validation, and add more supportive error message. Download the latest v1.25.0 binary, and install it with systemd service or just manually start the binary, will reproduce these errors. <details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions /close
Closing this issue. <detailsIn response to : kubelet 1.24+ is not working with the docker, unless using Are you using it? If not, try switching to other available container runtimes. Please re-open if you have further questions Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, I got to know the , so need to add runtime validation by the fix PR: PTAL thanks.


Question: SRCS=sampledeviceplugin ARCH ?= amd64 TARGET ?= $(CURDIR) <ins> GOARM ?= 7 </ins> GOLANG_VERSION ?= latest SRC_DIR = $(notdir $(shell pwd)) export

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Building the image binaries requires the GOARM variable to be set. Because it is not defined in all Makefiles, cannot the image binaries for the images in question. What you expected to happen: should succeed. How to reproduce it (as minimally and precisely as possible): git clone cd kubernetes/test/images/entrypoint-tester make Anything else we need to know?: Environment: N/A Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten