Question: return true } <del> func createSubnetMappings(subnetIDs []string) []*elbv2.SubnetMapping { </del> <ins> func createSubnetMappings(subnetIDs []string, allocationIDs []string) []*elbv2.SubnetMapping { </ins> response := []*elbv2.SubnetMapping{} <del> for _, id := range subnetIDs { // Ignore AllocationId for now response = append(response, &elbv2.SubnetMapping{SubnetId: aws.String(id)}) </del> <ins> for index, id := range subnetIDs { sm := &elbv2.SubnetMapping{SubnetId: aws.String(id)} if len(allocationIDs) > 0 { sm.AllocationId = aws.String(allocationIDs[index]) } response = append(response, sm) </ins> } return response

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws Anything else we need to know?: AWS NLBs support static IP addresses by associating an EIP per AZ. An already allocated EIP can be supplied with the . A user can then supply EIP allocation IDs in the service definition of kubernetes. By that the service will be available via those static IPs.
eip's for white listing are really important in the enterprise, we need this feature now, as a result we have to manually create nlb's, would really like to see this
This is also important for hosting providers, who might have huge numbers of apex domain DNS records pointed at EIPs. Associating an already-allocated EIP with a new Kubernetes-generated NLB would be huge!
This feature would save us weeks of work reimplementing the cloud controller just so we can preserve a special ELB that has static IP addresses... I really wish it was available
I'd like to contribute time to working on this. Is anyone else actively working on adding support?
That's awesome news. Thanks for working on this. Do you need assistance in testing this in an AWS environment? I might help out with that ...
Sure, the PR has been approved for testing so the binaries are available. I'm moving the code into our soak environment to get it some more exposure this week.
I've been trying with: But I only go a NLB without the EIPs deployed. Didn't find any errors in the logs. Do I have to enable a feature flag for it or kind of?
The reviewer found a typo where I may not be collecting the correct annotation. I'm waiting for the new build to complete and then rolling to our soak test. I'm passing through any AWS error and also error if the number of EIP and Subnets don't match. Let me get it online with the changes from code review and I'll dig in deeper. Also, are you using the binaries directly from the build system or just Kubernetes 1.12?
I checked with the fixes you provided yesterday. Works :) Created a service which spawns over 3AZs and supplied 3 EIP allocations. NLBs gets created correctly.
Can you also comment on the PR? I'm sure having multiple successful tests would be helpful to the reviewer.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hey, thanks for doing this, I was just looking for a way to associate one eip to a nlb, one question though why do you need one eip for each AZ? now the nlb creates a 1 public IP that balance to 3 AZ using traefik in my case when I deploy the service, thanks
Due to the fact that EIPs are represented by network hardware in AWS's datacenter you'd like to have redundant instances. If you run with only one EIP and the AZ of this EIP goes down, you won't be able to reach the (still up and running) traefik deployment in the other two AZs.
Hi, I've seen the PR () has been waiting for an update for three months. do you need a hand? do you want me to pick it up? I am really interested on this to be merged :smile:
Can you help write the tests, ? I'm completely lost there to be honest.
Hi I've found out that test-coverage on the aws provider is low and there is no tests on the so I can't find tests on NLB. I am going to open a new issue and work on improving coverage for that and hopefully I'll find out how to add tests for the new annotation. As you can see below current coverage for and is low and can't find tests for NLB:
Thanks for the help. I thought I was just missing being able to locate them.
+1 need this ASAP We are hosted on EKS so only official version supported. Any idea when this will be merged?
.... As a workaround, I'm just manually creating a second NLB (with EIPs) and changing the listeners to forward the traffic to the automatically created target groups (from the original svc-linked NLB).
Waiting for this as well on EKS. Thanks!
would be great to have that in place. thanks! I can confirm that the manual workaround by works fine, but it is hard to automate that flow.
+1 Waiting for this as well on EKS.

+1
I have created a node project that runs as a cronjob and monitors a SOURCE target group and clones the targets to a DESTINATION target group. So when you create a LoadBalancer of type 'nlb' in kubernetes, all you need to do is manually create a new 'nlb' in AWS Console, assign your EIP to your NLB for a specific 'subnet'(availability zone), then create a target group with no targets; use the ARN from this target group as the destination and use the ARN of the kubernetes target group as the SOURCE. More information here: Public docker image:
I have hard time getting this working with NLB using ingress controller : Even subnets are not taking effect here , its not passing my configurations in the API that creates the NLB kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: ingress-nginx ingress-nginx annotations: nlb "eipalloc-,eipalloc-,eipalloc-" "subnet-,subnet-" spec: type: LoadBalancer selector: ingress-nginx ingress-nginx ports: name: http port: 80 targetPort: http name: https port: 443 targetPort: https =======================================================``
I am also getting the same error can anyone help me with this ??
what version of k8s you are using ? this is not supported before 1.15 as far as I know so my issue was that I tried to implement on EKS
Hi i am using the latest version of k8 1.15 .5
I am not using EKS with my solution but the problem is the same as you mentioned
New to EKS. Could you elaborate how the connectivity from NLB that we have created manually (as you mentioned ) to the ingress-nginx pod?
Has it worked for you in k8s 1.15 EKS?
How to get Allocation IDs?
my elastic eip ids cannot be found at service creation, so it just sits in pending forever. if I resort to listing the subnets on their own annotation, (not using subnet discovery) same thing. no matter what I do it seems to not find me elastic ip addresses. Are you sure nlb type is supported to work with Elastic IPs?

From the AWS docs it seem they are pretty heavily trying to discourage use of the nlb type (Cloud Provider Load Balancer) in favor of pushing the new AWS Load balancer.
See available annotations: https://kubernetes- If you have done that: what do your aws-lb-controller logs tell you? I guess there’s an access problem with your iam role. Is your aws-lb-controller iam autorized to list and modify eip’s? Are the eip annotations exactly correct, can they be found by the controller? The controller’s log stream should have some kind of an error there. I have successfully assigned eips using service annotations to provision an aws nlb, but pay attention, you need to install the aws-lb-controller yourself. The kubernetes built-in aws library does not support eips.
Here is the annotation needed for assigning a static Elastic IP to your NLB using the AWS Load Balancer (Ingress) Controller https://kubernetes- New Controller link https://kubernetes-


Question: client = f.ClientSet Bootstrap(f) }) <del> ginkgo.AfterEach(func() { framework.Logf("Deleting all statefulset in namespace: %v", namespace) e2estatefulset.DeleteAllStatefulSets(client, namespace) }) </del> ginkgo.It("vsphere statefulset testing", func() { ginkgo.By("Creating StorageClass for Statefulset")

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The following issues are observed while running this e2e test case for vSphere CSI Migration feature validation. Issue-1 - test failed with Unexpected error: This is happening because YAML is using version. Issue-2 - After upgrading from . Test executed successfully but reported failure in the cleanup stage. when is called in DeleteAllStatefulSets, it does not find statefulset. moving call to in the defer block helps avoid printing this message. There may be a race between namespace deletion and after each execution. I am not familiar with recent e2e framework changes. What you expected to happen: Test should pass. How to reproduce it (as minimally and precisely as possible): execute using Kubernetes master branch builds. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere Cloud Provider
/sig testing /sig storage
/remove-sig testing The framework and CI are SIG testing. SIG storage owns their tests.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten


Question: return err } <ins> if timeout == 0 { timeout = kubectl.Timeout } r.Visit(func(info *resource.Info, err error) error { if err != nil { return err } return wait.PollImmediate(kubectl.Interval, timeout, func() (bool, error) { if err := info.Get(); !errors.IsNotFound(err) { return false, err } return true, nil }) }) </ins> r = resource.NewBuilder(mapper, typer, resource.ClientMapperFunc(f.UnstructuredClientForMapping), runtime.UnstructuredJSONScheme). Schema(schema). ContinueOnError().

When replacing an existing pod, the command fails as it tries to create a pod with a name that is still used by terminating pod. How does the option works? deletes a pod (e.g. mypod) creates a new pod with the same name (mypod) What happens? As the command tries to recreate the pod, it fails as the is in state. Thus it can not be created. The problem is the does not wait for a pod to be deleted. Reproducer: run mypod --image=yapei/hello-openshift --generator=run-pod/v1 get pod mypod -o yaml replace -f --force The last command ends with:
option introduce by
With set to positive value, what should be the default behaviour of the command? When set to , should the command hang for and then create the pod? Or is there a way to create a callback (inside a scheduler?) that creates the pod once the current one is deleted?
You could watch for the resource to be deleted and create after you see the deletion happen
have you ever tried to set , I think this will do a non graceful deletion. should make sure the old resource is deleted before it creates a new resource, if not, that's a bug.
Though the help says the is to be used only with , it returns the error above.
Then I think there are two issues we need to track: 1. option should be ignored or just returns error if users are not using option. 2. should always make sure the old resource has been deleted before create the new resource.


Question: "github.com/stretchr/testify/assert" <del> "k8s.io/component-base/metrics/legacyregistry" </del> <ins> compbasemetrics "k8s.io/component-base/metrics" </ins> runtimeapi "k8s.io/cri-api/pkg/apis/runtime/v1alpha2" "k8s.io/kubernetes/pkg/kubelet/metrics" ) func TestRecordOperation(t *testing.T) { <del> legacyregistry.MustRegister(metrics.RuntimeOperations) legacyregistry.MustRegister(metrics.RuntimeOperationsDuration) legacyregistry.MustRegister(metrics.RuntimeOperationsErrors) </del> <ins> // Use local registry var registry = compbasemetrics.NewKubeRegistry() var gather compbasemetrics.Gatherer = registry registry.MustRegister(metrics.RuntimeOperations) registry.MustRegister(metrics.RuntimeOperationsDuration) registry.MustRegister(metrics.RuntimeOperationsErrors) registry.Reset() </ins> l, err := net.Listen("tcp", "127.0.0.1:0") assert.NoError(t, err)

TestRecordOperation TestRecordOperation TestGetHistogramVecFromGatherer /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --/sig testing This may be related with
/assign I will do some initial researches on this. is similar
/sig instrument
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig instrument Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig instrumentation
/triage accepted /assign cc
To fix it gracefully, we may refactor the code in like The refactor would change a lot and need some confirmations or demo.(I prefer) A temporary workaround before refactoring it: is another try to ignore the dup error in but it is odd as should panic as designed if one register failed.
Three testings failed due to . Haven't found package promauto () being used in these testings. It might be other issues. These testings are: /filterwithonelabel
What are you guys doing here? What test is this? You cannot change the component-base metrics code base it is doing what it is supposed to be doing. Does this test suite involve multiple sequential runs of the same test?
I think this is odd to change the component-base metrics code base. But the refactor to use is another approach. You can try to reproduce it with is with
Oh, yes. This involves multiple sequential runs of the same test to . It uses
You can use to register the registry only once. It is important to reset the metrics, or next iteration in the test will fail
Thanks I other fixes in


Question: }, } <del> metrics := []string{ </del> <ins> metricNames := []string{ </ins> namespace + "_" + subsystem + "_automatic_reload_last_timestamp_seconds", } RegisterMetrics() for _, tc := range testCases { <ins> registry := testMetricsRegistry(t) </ins> encryptionConfigAutomaticReloadLastTimestampSeconds.Reset() encryptionConfigAutomaticReloadLastTimestampSeconds.WithLabelValues(tc.resultLabel, testAPIServerIDHash).Set(float64(tc.timestamp)) <del> if err := testutil.GatherAndCompare(legacyregistry.DefaultGatherer, strings.NewReader(tc.expectedValue), metrics...); err != nil { </del> <ins> if err := testutil.GatherAndCompare(registry, strings.NewReader(tc.expectedValue), metricNames...); err != nil { </ins> t.Fatal(err) } }

Change encryption config controller reload metrics to be consistent with other places (Authn, Authz) xref: Combine and into a single metric with with an enum value of and . /sig auth /triage accepted
cc
We probably should keep the existing metrics for some time to allow folks to migrate to the new one.
Ah I was thinking about this as well! Looking at "may be deleted at any time", I was leaning towards just removing it. If the common approach is to keep it for a release before removing, we could do that. I believe we can set a for the 2 metrics we want to remove.
I don't have a good sense of metrics stability because they tend to stay alpha forever and folks don't go back to update them. Since the associated feature set is GA, I think we should be conservative with the changes we make. Perhaps SIG instrumentation folks can provide some guidance here?
As part of the PR, I was going to ask when is the right time to graduate these metrics do you have thoughts on to take while deleting an alpha metric metrics should be graduated from alpha to beta and stable?


Question: package apiserver import ( <ins> "io/ioutil" </ins> "net/http" "net/http/httptest" "reflect"

I noticed that if I'm following a pod log that is being actively written to it will timeout after 5 minutes. However if the pod is not writing logs the request will not timeout. In that case the request will remain open seemingly indefinitely. I can change the behavior by modifying this line of code: To make it happen faster you can set the value lower. I've been testing with 1 minute. Steps to reproduce: cluster the nginx pod in another terminal generate log output by hitting the pod: (you'll need to change the ip). the logs: In this case whenever the WriteTimeout is reached curl will receive the zero sized chunk and the connection will be closed. If you leave out step 3 curl will continue seemingly indefinitely. This behavior definitely seems odd. Is the usage of WriteTimeout correct here? A quick search shows that others have been confused by the intention of WriteTimeout which is why I ask.
is this same as ?
From discussions on the OpenShift side it sounds somewhat related however what we're talking about is a reasonable (but arbitrary) timeout set. Users of are subjected to this timeout with no good way to pick up where they left off to avoid redownloading potentially large quantities of logs.
To me, it would make sense if the api endpoint had argument which would fetch logs from line (it's easy to track on client on which line we're at), exactly what does: I'm assuming that kubelet gets logs using docker api, unfortunately there is no such thing in docker's api:
I think users that want to do this should get the logs from elastic search instead of from kubectl.
I can't say I disagree about needing something like elastic search in a real world deployment. I guess part of the weirdness is that clients of the logging API rarely know how much is going to be pulled down before they make the request. I guess the answer now if you're following the logs is "5 minutes worth". I know this isn't how it's implemented today but in my mind following the logs should behave more like or . Meaning, when you state you want to follow you don't fetch the entire log file you just get wherever it happens to be (and maybe a little more for context). If you want the whole file you don't pass in .
Any update? This is really annoying and I'm thinking of patching my k8s.

I think the default API timeout has been lengthened to 60 minutes Defined: Used for API server: It looks like the kubelet timeout is still 5 minutes, so that would gate log streaming to 5 minutes:
More nuances in Looks like the API http server times out after 60 minutes: Watches time out between 5 and 10 minutes: Kubelet http server still times out after 5 minutes
Which means that getting logs timeouts after 5 minutes?
yeah
did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
nm, just merged the pr. did intentionally leave the kubelet timeout at 5 minutes, or should it be using the same ReadWriteTimeout as the api server?
we could easily bump up the kubelet timeout. The handlers on the kubelet don't have their own built in timeouts (like I did for watch), so a potential downside is long hanging /stats requests could hog connections, and depending on why they're hanging, continue consuming kubelet/cpu/cadvisor time. had a suggestion previously about using elasticsearch, are you running with a cluster that doesn't have the addons (, if you set it up with kube-up it should)?
IMO there's no reason not to make kubelet server's timeout really long, at least for these requests.
I'm trying to put in a fix to increase the kubelet timeouts to 1 hour to match the master. Unfortunately adding proper timeout handling for these cases may not make 1.0, but hopefully this can make it less annoying.


Question: MountPropagation:                            {Default: false, PreRelease: utilfeature.Alpha}, ExpandPersistentVolumes:                     {Default: false, PreRelease: utilfeature.Alpha}, CPUManager:                                  {Default: false, PreRelease: utilfeature.Alpha}, <ins> ServiceNodeExclusion:                        {Default: false, PreRelease: utilfeature.Alpha}, </ins> // inherited features from generic apiserver, relisted here to get a conflict if it is changed // unintentionally on either side:

For various reasons (network topology, heterogenous nodes, etc) a user may want to control the addition of a node to a cloud load balancer. A specific issue with the Azure Container Instances connector which introduces a virtual node can be found here: The right way to do this is via a node annotation, e.g.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen This is still a problem.. any word on when feature gate goes out of alpha?
Setup the enhancement tracking issue for this Still discussing with sig-network on how we want to move forward with this.


Question: if err != nil { return false, matchingPodExists, err } <del> if priorityutil.NodesHaveSameTopologyKey(node, existingPodNode, term.TopologyKey) { </del> <ins> if priorityutil.NodesHaveSameTopologyKey(nodeInfo.Node(), existingPodNode, term.TopologyKey) { </ins> return true, matchingPodExists, nil } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --It is not a secret that . In my opinion the main reason for the slowness is the flexibility of the inter-pod affinity/anti-affinity rules. However, in most practical scenarios users do not need all the flexibility. For example, people often do not use arbitrary labels in topology key of the affinity rules. They often use node, zone, or region as the topology. Node in particular is used often. That's why I am proposing a fast path for the case that topology key is Node. Today, the affinity/anti-affinity predicate does not have any distinction for topology keys and in the cluster regardless. The logic should recognize "" and scan only pods on the same node for this topology key. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature /sig scheduling cc/
ref/
/assign
GitHub didn't allow me to assign the following users: misterikkit. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
We found some other simple optimizations that had a significant impact on the affinity predicate. I will upload PRs for those as well.
Reopening this as the performance of anti-affinity still doesn't seem to be satisfactory enough on large clusters (i.e 5k nodes) - and it's causing them to fail to startup. Ref: issue I can help with evaluating any performance improvements if needed.
The change sent improved performance of the predicate, but apparently it is not enough to make our large scale scalability tests happy.
This issue is not about general performance improvment of affinity/anti-affinity. It was to implement a particular improvment which is already implemented. There are other performance improvement issues. I am going to close this one.
Sounds good. Thanks.
How come with kops 1.10.0 I still see all kube-dns pods on the same node because the kube-dns deployment doesn't specify any anti-affinity rule?
I think you are correct. you can double-check with the option to . I have a v1.9 cluster in front of me, and kube-dns has no affinity rules. (I didn't use kops though)


Question: // See http://docs.openstack.org/user-guide/cli_config_drive.html type Metadata struct { Uuid             string           `json:"uuid"` <del> Name             string           `json:"name"` </del> <ins> Hostname         string           `json:"hostname"` </ins> AvailabilityZone string           `json:"availability_zone"` Devices          []DeviceMetadata `json:"devices,omitempty"` // .. and other fields we don't care about.  Expand as necessary.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The OpenStack cloud provider currently uses the name of the kubelet node to register with the API server. However the name is not guaranteed to conform to the DNS-1123 subdomain spec. Given a node name of , the following message appears in the kubelet logs: What you expected to happen: The OpenStack Cloud provider uses the instance id instead of the instance name. This will guarantee that the id conforms the DNS-1123 spec without making any assumptions on what the instance name may be. How to reproduce it (as minimally and precisely as possible): In the OpenStack console, rename a kubelet node to and attempt to register the kubelet node to the api server. Anything else we need to know?: Environment: OpenStack Newton Kubernetes version (use ):Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 14.04.5 Kernel (e.g. ): Linux -cab9-44ff-9c99- 4.4.0-103-generic ~14.04.1-Ubuntu SMP Mon Dec 4 19:33:04 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others:
/sig openstack
/assign I prefer to conform hostname to DNS-1123 subdomain spec. Already submitted a PR on this.
I think the issue may be more than just the hostname. The hostname on my vm is set to Since the OpenStack cloud provider uses the instance name to register with the api server, I also get error messages like the following: It seems like for OpenStack specifically, it sets it's node name to the name returned from the OpenStack metadata server () which is not restricted to being DNS-1123 compliant.
The node name must match ([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9], not contain '/'.
I can confirm that Openstack can boot up instances with the name like . But this will break kubelet currently as stated above. For such cloud provider based nodes, the nodeName registered is retrieved from the provider, which should not be allowed when registering nodes for kubelet.

Agree. +1. I've an extra check in PR . PTAL. Thanks.
I think the question here is: Why did you choose to use as the nodename? This is the display name of the server, a property made for humans by humans. Essentially it is a field that everybody should be able to fill in whatever they think their server should be called. This doesn't need to be unique and has no guarantees about format or length or character format. We've had a number of people putting in unicode characters in the display name, for example. Furthermore, it can be changed during runtime of the server, without rebooting. Other possibilities for the nodename are: instance UUID, which is a string in a defined format, guaranteed to be unique and is only separated by dashes. Found in If you would like to have some way to associate your nodename with whatever is shown in the Horizon dashboard as server display name, there is . OpenStack already takes care to sanitize any weird display name. E.g. some weird display name like gets a reasonable hostname: <img width="745" alt="screen shot 2018-01-10 at 08 52 49" src="https://user-"What do you think about changing the source for selecting the nodename?
cc
Once upon a time, the cloud provider's (only!) role was to discover a dynamic list of nodes by regex. In that world, using the nodename made perfect sense because the admin could then configure it to be something like , with the obvious use of regex patterns to group the relevant nodes. Fast forward a few years, and this style of node registration is long gone, and there is no need to make the node name human readable, and in fact numerous reasons to choose other unique identifiers. There is still to removing "human-friendly" node names, however. In the end, I didn't see a good alternative that would please everyone (and meet the required technical character set/length constraints). I stuck with because at least it was under the control of the cluster admin, and so it could be obviously contrived to match our imposed k8s node name == openstack instance name constraint (once that constraint was communicated).


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" <ins> testapigroupv1 "k8s.io/apimachinery/pkg/apis/testapigroup/v1" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/runtime/serializer"

What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "type": "integer", "paramType": "query", "name": "timeoutSeconds", <del> "description": "Timeout for the list/watch call.", </del> <ins> "description": "Timeout for the list/watch call. This limits the duration of the call, regardless of any activity or inactivity.", </ins> "required": false, "allowMultiple": false },

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Watches done with the field of set to some value T end after T regardless of whether there has been any activity in the meantime. What you expected to happen: I expected the watch to end when no events have been delivered in the last T. In ordinary usage, a "timeout" is something that happens if there has been no activity for the specified length of time. How to reproduce it (as minimally and precisely as possible): Create a configmap named in namespace . Then in one shell thrash it periodically like this: Observe that it proceeds to update the configmap every 7 seconds or so. In another shell do a watch with a timeout; observe that it ends even though the thrashing continues. Anything else we need to know?: Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.4+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-11-21T17:22:25Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}Cloud provider or hardware configuration: local hardware OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): Linux n17s2ctrl 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: local Others:
I am told the observed behavior is the intended behavior, the problem here is that the field name and comment are misleading.
/sig api-machinery
It's a timeout on the request lifetime, not an inactivity timeout
The inactivity condition is implied in ordinary usage of the word "timeout". A word like "lifetime", "duration", or "time limit" would be a better description of the existing behavior. I do not propose changing the field name for this, that would be more work than it is worth. I propose updating the commentary on that field to make its meaning clear. BTW, what exactly is this field's meaning to a list operation?


Question: defer m.wg.Done() m.server.Serve(sock) }() <del> _, conn, err := dial(m.socket) if err != nil { return err </del> <ins> var lastDialErr error wait.PollImmediate(1*time.Second, 10*time.Second, func() (bool, error) { var conn *grpc.ClientConn _, conn, lastDialErr = dial(m.socket) if lastDialErr != nil { return false, nil } conn.Close() return true, nil }) if lastDialErr != nil { return lastDialErr </ins> } <del> conn.Close() </del> klog.Infof("Starting to serve on %v", m.socket) return nil

Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: TestDevicePluginReRegistrationProbeMode Testgrid link: Reason for failure: Anything else we need to know: Reproduce with /sig node
/assign
See
not sure this is actually fixed, I see a failure from today


Question: client := f.ClientSet // Add a unique label to the namespace <del> ns, err := client.CoreV1().Namespaces().Get(context.TODO(), namespace, metav1.GetOptions{}) framework.ExpectNoError(err, "error getting namespace %s", namespace) if ns.Labels == nil { ns.Labels = map[string]string{} } ns.Labels[f.UniqueName] = "true" _, err = client.CoreV1().Namespaces().Update(context.TODO(), ns, metav1.UpdateOptions{}) </del> <ins> nsPatch, err := json.Marshal(map[string]interface{}{ "metadata": map[string]interface{}{ "labels": map[string]string{f.UniqueName: "true"}, }, }) framework.ExpectNoError(err, "error marshaling namespace %s", namespace) _, err = client.CoreV1().Namespaces().Patch(context.TODO(), namespace, types.StrategicMergePatchType, nsPatch, metav1.PatchOptions{}) </ins> framework.ExpectNoError(err, "error labeling namespace %s", namespace) }

In VMware Tanzu Supervisor Cluster k8s setup, we observed that the AdmissionWebhook tests fail intermittently while updating the namespace with the error: "the object has been modified; please apply your changes to the latest version and try again". This is because the update conflicts with an update being performed by another controller. The test could use Patch while adding labels via types.StrategicMergePatchType to prevent the conflict. /sig testing /sig api-machinery
(sig testing owns e.g. the frameworks but API Machinery own their test cases)
/triage accepted


Question: <ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ /* Package cache implements a data structure used by the attach/detach controller to keep track of volumes, the nodes they are attached to, and the pods that reference them. It is thread-safe. */ package cache import ( "fmt" "sync" ) // AttachDetachVolumeCache defines the set of operations the volume cache // supports. type AttachDetachVolumeCache interface { // AddVolume adds the given volume to the list of volumes managed by the // attach detach controller. // If the volume already exists, this is a no-op. AddVolume(volumeName string) // AddNode adds the given node to the list of nodes the specified volume is // attached to. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. // If the node already exists for the specified volume, this is a no-op. AddNode(nodeName, volumeName string) error // AddPod adds the given pod to the list of pods that are scheduled to // the specified node and referencing the specified volume. // If no node with the name nodeName exists in the list of attached nodes, // an error is returned. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. // If the pod already exists for the specified volume, this is a no-op. AddPod(podName, nodeName, volumeName string) error // DeleteVolume removes the given volume from the list of volumes managed // by the attach detach controller. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. // All attachedNodes must be deleted from the volume before it is deleted. // If the specified volume contains 1 or more attachedNodes, an error is // returned. DeleteVolume(volumeName string) error // DeleteNode removes the given node from the list of nodes the specified // volume is attached to. // If no node with the name nodeName exists in the list of attached nodes, // an error is returned. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. // All scheduledPods must be deleted from the node before it is deleted. // If the specified node contains 1 or more scheduledPods, an error is // returned. DeleteNode(nodeName, volumeName string) error // DeletePod removes the given pod from the list of pods that are scheduled // to the specified node and referencing the specified volume. // If no pod with the name podName exists for the specified volume/node, an // error is returned. // If no node with the name nodeName exists in the list of attached nodes, // an error is returned. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. DeletePod(podName, nodeName, volumeName string) error // VolumeExists returns true if the volume with the specified name exists // in the list of volumes managed by the attach detach controller. VolumeExists(volumeName string) bool // NodeExists returns true if the node with the specified name exists in // the list of nodes the specified volume is attached to. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. NodeExists(nodeName, volumeName string) (bool, error) // PodExists returns true if the pod with the specified name exists in the // list of pods that are scheduled to the specified node and referencing // the specified volume. // If no node with the name nodeName exists in the list of attached nodes, // an error is returned. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. PodExists(podName, nodeName, volumeName string) (bool, error) } // NewAttachDetachVolumeCache returns a new instance of the // AttachDetachVolumeCache. func NewAttachDetachVolumeCache() AttachDetachVolumeCache { return &attachDetachVolumeCache{ volumesManaged: make(map[string]volume), } } type attachDetachVolumeCache struct { // volumesManaged is a map containing the set of volumes managed by the // attach/detach controller. The key in this map is the name of the unique // volume identifier and the value is a volume object containing more // information about the volume. volumesManaged map[string]volume sync.RWMutex } // The volume object represents a volume that is being tracked by the attach // detach controller. type volume struct { // name contains the unique identifer for this volume. name string // attachedNodes is a map containing the set of nodes this volume has // succesfully been attached to. The key in this map is the name of the // node and the value is a node object containing more information about // the node. attachedNodes map[string]node } // The node object represents a node that a volume is attached to. type node struct { // name contains the name of this node. name string // scheduledPods is a map containing the set of pods that are scheduled to // this node and referencing the underlying volume. The key in the map is // the name of the pod and the value is a pod object containing more // information about the pod. scheduledPods map[string]pod } // The pod object represents a pod that is scheduled to a node and referncing // the underlying volume. type pod struct { // name contains the name of this pod. name string } // AddVolume adds the given volume to the list of volumes managed by the attach // detach controller. // If the volume already exists, this is a no-op. func (vc *attachDetachVolumeCache) AddVolume(volumeName string) { vc.Lock() defer vc.Unlock() if _, exists := vc.volumesManaged[volumeName]; !exists { vc.volumesManaged[volumeName] = volume{ name:          volumeName, attachedNodes: make(map[string]node), } } } // AddNode adds the given node to the list of nodes the specified volume is // attached to. // If no volume with the name volumeName exists in the list of managed volumes, // an error is returned. // If the node already exists for the specified volume, this is a no-op. func (vc *attachDetachVolumeCache) AddNode(nodeName, volumeName string) error { vc.Lock() defer vc.Unlock() vol, volExists := vc.volumesManaged[volumeName] if !volExists { return fmt.Errorf( "failed to add node %q to volume %q--no volume with that name exists in the list of managed volumes", nodeName, volumeName) } if _, nodeExists := vol.attachedNodes[nodeName]; !nodeExists { vc.volumesManaged[volumeName].attachedNodes[nodeName] = node{ name:          nodeName, scheduledPods: make(map[string]pod), } } return nil } // AddPod adds the given pod to the list of pods that are scheduled to the // specified node and referencing the specified volume. // If no node with the name nodeName exists in the list of attached nodes, // an error is returned. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. // If the pod already exists for the specified volume, this is a no-op. func (vc *attachDetachVolumeCache) AddPod(podName, nodeName, volumeName string) error { vc.Lock() defer vc.Unlock() volObj, volExists := vc.volumesManaged[volumeName] if !volExists { return fmt.Errorf( "failed to add pod %q to node %q volume %q--no volume with that name exists in the list of managed volumes", podName, nodeName, volumeName) } nodeObj, nodeExists := volObj.attachedNodes[nodeName] if !nodeExists { return fmt.Errorf( "failed to add pod %q to node %q volume %q--no node with that name exists in the list of attached nodes for that volume", podName, nodeName, volumeName) } if _, podExists := nodeObj.scheduledPods[podName]; !podExists { vc.volumesManaged[volumeName].attachedNodes[nodeName].scheduledPods[podName] = pod{ name: podName, } } return nil } // DeleteVolume removes the given volume from the list of volumes managed by // the attach detach controller. // If no volume with the name volumeName exists in the list of managed volumes, // an error is returned. // All attachedNodes must be deleted from the volume before it is deleted. // If the specified volume contains 1 or more attachedNodes, an error is // returned. func (vc *attachDetachVolumeCache) DeleteVolume(volumeName string) error { vc.Lock() defer vc.Unlock() volObj, volExists := vc.volumesManaged[volumeName] if !volExists { return fmt.Errorf( "failed to delete volume %q--no volume with that name exists in the list of managed volumes", volumeName) } if len(volObj.attachedNodes) > 0 { return fmt.Errorf( "failed to remove volume %q from list of managed volumes--the volume still contains %v nodes in its list of attached nodes", volumeName, len(volObj.attachedNodes)) } delete( vc.volumesManaged, volumeName) return nil } // DeleteNode removes the given node from the list of nodes the specified // volume is attached to. // If no node with the name nodeName exists in the list of attached nodes, an // error is returned. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. // All scheduledPods must be deleted from the node before it is deleted. // If the specified node contains 1 or more scheduledPods, an error is // returned. func (vc *attachDetachVolumeCache) DeleteNode(nodeName, volumeName string) error { vc.Lock() defer vc.Unlock() volObj, volExists := vc.volumesManaged[volumeName] if !volExists { return fmt.Errorf( "failed to delete node %q from volume %q--no volume with that name exists in the list of managed volumes", nodeName, volumeName) } nodeObj, nodeExists := volObj.attachedNodes[nodeName] if !nodeExists { return fmt.Errorf( "failed to delete node %q from volume %q--no node with the that name exists in the list of attached nodes for that volume", nodeName, volumeName) } if len(nodeObj.scheduledPods) > 0 { return fmt.Errorf( "failed to remove node %q from volume %q--the node still contains %v pods in its list of scheduled pods", nodeName, volumeName, len(nodeObj.scheduledPods)) } delete( vc.volumesManaged[volumeName].attachedNodes, nodeName) return nil } // DeletePod removes the given pod from the list of pods that are scheduled // to the specified node and referencing the specified volume. // If no pod with the name podName exists for the specified volume/node, an // error is returned. // If no node with the name nodeName exists in the list of attached nodes, // an error is returned. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. func (vc *attachDetachVolumeCache) DeletePod(podName, nodeName, volumeName string) error { vc.Lock() defer vc.Unlock() volObj, volExists := vc.volumesManaged[volumeName] if !volExists { return fmt.Errorf( "failed to delete pod %q from node %q volume %q--no volume with that name exists in the list of managed volumes", podName, nodeName, volumeName) } nodeObj, nodeExists := volObj.attachedNodes[nodeName] if !nodeExists { return fmt.Errorf( "failed to delete pod %q from node %q volume %q--no node with that name exists in the list of attached nodes for that volume", podName, nodeName, volumeName) } if _, podExists := nodeObj.scheduledPods[podName]; !podExists { return fmt.Errorf( "failed to delete pod %q from node %q volume %q--no pod with that name exists in the list of scheduled pods under that node/volume", podName, nodeName, volumeName) } delete( vc.volumesManaged[volumeName].attachedNodes[nodeName].scheduledPods, podName) return nil } // VolumeExists returns true if the volume with the specified name exists in // the list of volumes managed by the attach detach controller. func (vc *attachDetachVolumeCache) VolumeExists(volumeName string) bool { vc.RLock() defer vc.RUnlock() _, volExists := vc.volumesManaged[volumeName] return volExists } // NodeExists returns true if the node with the specified name exists in the // list of nodes the specified volume is attached to. // If no volume with the name volumeName exists in the list of managed // volumes, an error is returned. func (vc *attachDetachVolumeCache) NodeExists(nodeName, volumeName string) (bool, error) { vc.RLock() defer vc.RUnlock() volObj, volExists := vc.volumesManaged[volumeName] if !volExists { return false, fmt.Errorf( "failed to check if node %q exists under volume %q--no volume with that name exists in the list of managed volumes", nodeName, volumeName) } _, nodeExists := volObj.attachedNodes[nodeName] return nodeExists, nil } // PodExists returns true if the pod with the specified name exists in the list // of pods that are scheduled to the specified node and referencing the // specified volume. // If no node with the name nodeName exists in the list of attached nodes, an // error is returned. // If no volume with the name volumeName exists in the list of managed volumes, // an error is returned. func (vc *attachDetachVolumeCache) PodExists(podName, nodeName, volumeName string) (bool, error) { vc.RLock() defer vc.RUnlock() volObj, volExists := vc.volumesManaged[volumeName] if !volExists { return false, fmt.Errorf( "failed to check if node %q exists under volume %q--no volume with that name exists in the list of managed volumes", nodeName, volumeName) } nodeObj, nodeExists := volObj.attachedNodes[nodeName] if !nodeExists { return false, fmt.Errorf( "failed to check if pod %q exists under node %q volume %q--no node with that name exists in the list of attached nodes for that volume", podName, nodeName, volumeName) } _, podExists := nodeObj.scheduledPods[podName] return podExists, nil } </ins>

e2e-cos-alpha-features All I'm not sure it was ever running. No response No response /sig node /priority important-longterm since it's alpha features, setting priority lower
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
kubelet log shows error below:
We tried to fix it in before. I opened


Question: f.NamespacePodSecurityLevel = admissionapi.LevelPrivileged m := newMockDriverSetup(f) ginkgo.Context("CSI workload information using mock driver", func() { <del> var ( podInfoTrue  = true podInfoFalse = false ) </del> tests := []struct { name                   string <del> podInfoOnMount         *bool </del> <ins> podInfoOnMount         bool </ins> deployClusterRegistrar bool expectPodInfo          bool expectEphemeral        bool }{ { <del> name:                   "should not be passed when podInfoOnMount=nil", podInfoOnMount:         nil, deployClusterRegistrar: true, expectPodInfo:          false, expectEphemeral:        false, }, { </del> name:                   "should be passed when podInfoOnMount=true", <del> podInfoOnMount:         &podInfoTrue, </del> <ins> podInfoOnMount:         true, </ins> deployClusterRegistrar: true, expectPodInfo:          true, expectEphemeral:        false, }, { name:                   "contain ephemeral=true when using inline volume", <del> podInfoOnMount:         &podInfoTrue, </del> <ins> podInfoOnMount:         true, </ins> deployClusterRegistrar: true, expectPodInfo:          true, expectEphemeral:        true, }, { name:                   "should not be passed when podInfoOnMount=false", <del> podInfoOnMount:         &podInfoFalse, </del> <ins> podInfoOnMount:         false, </ins> deployClusterRegistrar: true, expectPodInfo:          false, expectEphemeral:        false,

1 ci-kubernetes-e2e-gci-gce 1 ci-cos-containerd-e2e-cos-gce in the master blocking test grid 1 ci-kubernetes-e2e-gci-gce-network-proxy-grpc 1 ci-kubernetes-e2e-gci-gce-proto Kubernetes e2e suite.[It] [sig-storage] CSI Mock workload info CSI PodInfoOnMount Update should be passed when update from false to true 12/14 code thaw after 1.29.0 release was merged at the time and is related to PodInfoOnMount. /cc /sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
testCase: podInfoOnMount should be passed when update from false to true kubelet log: csi-mockplugin-0-mock log: the function in the test case is not working as expected. expect log at Line 2, but actual log is at Line 3.
is called twice for the same volume sometime even though the volume is already published successfully. Not only this issue, but also other issues like I wonder if it is a potential bug or a normal behavior. cc what do you think?
good catch. I think that any test that depends on exact index of NodePublish call will be flaky, sometimes Kubelet has to retry NodePublish. It would be better to check for NodePublish parameter changes. For example, test should check there were 1 or more NodePublish calls with no pod info and then 1 or more NodePublish calls with pod info afterwards.


Question: break } } <ins> if dswp.hasAddedPods { t.Fatalf("HasAddedPod should be false but it is true") } </ins> if !found { t.Fatalf( "Could not found pod volume %v in the list of actual state of world volumes to mount.",

We observed that while a container is in the process of being torn down, but not completed, the volume gets unmounted and then subsequently detached, causing the fs to enter read-only mode. This happened after kubelet restarted. We observed that on the restart, the volume was not to DSW because the Pod was marked for deletion. In the normal case, we only remove the pod from DSW when the containers have finished teardown: However, in the reconstruction case, it looks like if a pod has started termination, but not necessarily completed, then we don't add the pod to DSW: We probably need to be consistent and change the add pod criteria to also add even if it's started terminating. The side effect is that in the normal case we may do unnecessary mounts if a pod is quickly created and deleted. We can also consider a flag to only use the stricter check during reconstruction. What happened in our env was that kubelet was crash looping, and then all the pods on the node were deleted. It looks like there was a good 30 seconds between volume unmount and container teardown. I think this may be easily reproducible if the pod has a long graceful termination period and the container ignores the stop signal. No response <details22 </details<details</details<details</details<details</details<details</details<details</details
/triage accepted
Yeah this looks problematic. What was the exact kube version btw? So it was possible that container is still writing to the volume? Do we know why fs enters read-only mode when this happens? I know the behaviour of kubelet is problematic regardless, but still. I think adding volumes belonging to terminating pod doesn't sound so bad. I wonder if we could change this to .
Does this mean instead of checking ShouldPodContainersBeTerminating to decide if we add the pod or not, we should check IsPodTerminationRequested or not to decide? Or we should use the same check ShouldPodRuntimeBeRemoved with reconstruction(this one is more consistent)? pointed out that there were already scaling challenges with volume manager processing pods it doesn't need to.
For reconstruction, we need to add the volume in DSW if there is a chance that containers are running. So is the best call to determine that.
We observed a similar issue but could only reproduce it with emptydir volumes. Do you think is the same issue?

/assign sunnylovestiramisu
thanks for the report. I think this issue could impact emptydir volumes too.
Is there any recommended way to make a pod having a long graceful termination period? Tried with some simple pods but it just terminated right away.
You can use the sample pod and instructions from here
cc
Here is my repro pod and steps: the pod shell, another window (window 2), ssh to the node another window (window 3), window 2, in window 1, exec back into the pod . The file will be gone. try . It will return When looking at kubelet logs, you'll see the log messages that indicate the volume was reconstructed without being to dsw. and UnmountVolume happen soon after: Pod lifecycle logs:
Tested locally via kind cluster. /k master is showing error of touch: cannot touch '/scratch/test': No such file or directory. cluster with the fix in this pr shows no error. And running shows no result in the log.


Question: # If true, current user will be added to the docker group on test node SETUP_NODE=false # If true QoS Cgroup Hierarchy is created and tests specifc to the cgroup hierarchy run <del> TEST_ARGS=--cgroups-per-qos=true </del> <ins> TEST_ARGS=--cgroups-per-qos=false </ins>

In node-e2e-test on Containervm image, the containers of the pods created by Kubernetes are subcontainers of docker-daemon/docker/. It means that the runtime resource usage (docker-daemon) is not correct, because all pods are included. This problem maybe due to container misconfiguration in node-e2e-test.
The kubelet on debian-based containervm image is misconfigured. Will fix it shortly.


Question: if !apiequality.Semantic.DeepEqual(mungedPodSpec, oldPod.Spec) { // This diff isn't perfect, but it's a helluva lot better an "I'm not going to tell you what the difference is". //TODO: Pinpoint the specific field that causes the invalid error after we have strategic merge diff <del> specDiff := diff.ObjectDiff(mungedPodSpec, oldPod.Spec) </del> <ins> specDiff := cmp.Diff(oldPod.Spec, mungedPodSpec) </ins> allErrs = append(allErrs, field.Forbidden(specPath, fmt.Sprintf("pod updates may not change fields other than `spec.containers[*].image`, `spec.initContainers[*].image`, `spec.activeDeadlineSeconds`, `spec.tolerations` (only additions to existing tolerations) or `spec.terminationGracePeriodSeconds` (allow it to be set to 1 if it was previously negative)n%v", specDiff))) }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --pod with nodeSeletor edit pod, delete the nodeSeletor field report error with: ! The errors message should displayed as: Because the nodeSeletor is been deleted with this edit. Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig apps
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle rotten


Question: trap "teardown" EXIT <del> echo "WARNING: this test is a no op that only attempts to launch guestbook resources." </del> # Launch the guestbook example ${KUBECTL} create -f "${GUESTBOOK}" <del> sleep 15 POD_LIST_1=$(${KUBECTL} get pods -o template '--template={{range.items}}{{.id}} {{end}}') echo "Pods running: ${POD_LIST_1}" # TODO make this an actual test. Open up a firewall and use curl to post and # read a message via the frontend ${KUBECTL} stop -f "${GUESTBOOK}" POD_LIST_2=$(${KUBECTL} get pods -o template '--template={{range.items}}{{.id}} {{end}}') echo "Pods running after shutdown: ${POD_LIST_2}" </del> <ins> # Verify that all pods are running wait_for_running if [[ "${KUBERNETES_PROVIDER}" == "gce" ]]; then gcloud compute firewall-rules create --allow=tcp:8000 --network="${NETWORK}" --target-tags="${MINION_TAG}" guestbook-e2e-minion-8000 fi # Add a new entry to the guestbook and verify that it was remembered frontend_addr=$(${KUBECTL} get service frontend -o template '--template={{range .publicIPs}}{{.}}{{end}}:{{.port}}') echo "Waiting for frontend to serve content" serving=0 for i in {1..12}; do entry=$(curl "http://${frontend_addr}/index.php?cmd=get&key=messages") || true echo ${entry} if [[ "${entry}" == '{"data": ""}' ]]; then serving=1 break fi sleep 10 done if [[ "${serving}" == 0 ]]; then echo "Pods did not start serving content in time" exit 1 fi curl "http://${frontend_addr}/index.php?cmd=set&key=messages&value=TestEntry" entry=$(curl "http://${frontend_addr}/index.php?cmd=get&key=messages") if [[ "${entry}" != '{"data": "TestEntry"}' ]]; then echo "Wrong entry received: ${entry}" exit 1 fi </ins> exit 0

Part of issue . Re-write test in Go, generate unique names for top level resources, ensure test is idempotent, clean up all resources at end of test.
What should we do about this test? The current test in only brings up the guestbook but doesn't even confirm that it is up or tries to connect to any of the ports... Was the point of that test case to test /? Does it make sense to rewrite this test in Go or do we already have more test coverage from e.g. ? If it makes sense to rewrite it in Go, what should we accomplish with this test?
How about a functional test that inserts some names and tries to retrieve them i.e. enough to know that the service has basic functionality?
+1 to an actual functional test. The service is basic enough, it should be straightforward to do the same POST/GET from go that the js on the guestbook homepage does.
and I talked about this and for a while and think that they should remain as tests. This test needs to be improved: it needs to be turned into an actual test (we don't do anything with the service).
has not passed on gce or gke-ci since around 11am this morning (around when this PR was merged). Can you look at the Jenkins logs and try to figure out what is going wrong?
The problem is with guestbook itself not with the test. The frontend cannot connect to redis data base so it returns error response to the user. I created issue for that
I have been 100% absorbed for weeks with my own e2e test so I've not paid much attention to this. Does the test connect to the service via an external load balancer?
Nope. It's using proxy. For example:
Is this a reasonable use of the proxy service? Today told me that the proxy service is fine for contacting cluster-level services (note the dash Brian) but for regular applications we should use some other mechanism (e.g. external load balancer). ?
My first idea was to use ELB but it requires to create also firewall-rules using. I'm not sure if we want to do it.
PR has been reverted by The test is passing for me locally. I'll updated . Is there any chance that using proxy instead of ELB can cause the problem?

Personally, I think -- for now -- using an ELB is fine -- by setting the use external load balancer field in the System specification. I am working on a PR to fix an issue relating to the use of application endpoints using the proxy server (to address issue -- which can bite you if one of the fields of a JSON response contains a field called ).
It sounds to me like there are 2 issues: should the guestbook example make the guestbook app accessible to users? should the test test correctness? I think these are 2 totally different things. The right answer to (1) is clearly external IPs. That's what we want to teach users to do. (2) likely doesn't have the same requirements. Conflating the 2 cases might not be helpful. However, if you want to do that, use the external load balancer, not the proxy.
Yes, I am torn about this too. Another thing to consider is that a test that uses an external load balancer is going to take longer because it takes much longer to create a service with an external load balancer (almost instant with ELB vs. perhaps up to a minute with a load balancer) and then there is a teardown step as well when you delete the service (which deletes the forwarding rules etc.). As a point of principle we should decide if it is an approved use of the proxy mechansim for use in tests in order to make them more lightweight vs. using an ELB.
So there is agreement to use ELB. What about firewall rule? Port 8000 should be opened, I can do it by using gcloud, however it's not provider agnostic operation. Is there any other option for that?
Sadly not: I suggest just using gcloud / making the API calls for the firewall and keep this as a GCE specific test. Which is indeed a shame.
Reopening since the PR was reverted in


Question: nodeLeaseRenewIntervalFraction = 0.25 ) <ins> var etcHostsPath = getContainerEtcHostsPath() func getContainerEtcHostsPath() string { if sysruntime.GOOS == "windows" { return windowsEtcHostsPath } return linuxEtcHostsPath } </ins> // SyncHandler is an interface implemented by Kubelet, for testability type SyncHandler interface { HandlePodAdditions(pods []*v1.Pod)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig windows What happened: When deploying Kubernetes pods, the Linux containers will contain the following entries in the file: # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.240.0.67 dns-querier-1.dns-test-service.e2e-tests-dns- dns-querier-1 However, these are not set on Windows containers; the container's file does not contain any entry. This can be attributed to the fact that individual files cannot be mounted on Windows containers the same way they are on Linux containers. It can be fixed with a change to the kubelet once Windows Server 2019 has shipped, and once CRI-ContainerD is enabled on Windows. We cannot fix it for Windows Server 1803 or earlier with dockershim. What you expected to happen: The file should be managed and populated by Kubelet. How to reproduce it (as minimally and precisely as possible): Spawn a Windows pod, and check its container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cmd /S /C type C:WindowsSystem32driversetchosts Compare the result with a Linux pod's container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cat /etc/hosts Anything else we need to know?: Environment: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: N/A OS (e.g.

from /etc/os-release): Windows Server 1803 (10.0..48) Install tools: acs-engine v0.24.0 Others:
This requires single-file mapping, which won't work in Windows Server version 1803. It should be possible with Windows Server 2019 along with ContainerD
Will it work with Docker on Windows Server 2019?
Not in 18.09 or earlier. I'm not sure what release it will be in because it will require migrating to containerd.
should this issue be tracked for 1.13 in addition to If so please apply the appropriate priority and milestone labels please.
This one needs to be in v1.14. We need to get cri-containerd working first to fix this.
Thanks /milestone v1.14
By default, hosts file are not being used in Windows, it is all registry based. What features is this blocking? CC
The hosts file entries are still taken into account on Windows when it has to resolve a DNS name. This is blocking DNS conformance tests, like.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
Hello! 14 code freeze is coming in about 3 days, is this intended to be implemented in the following week?
No. It is meant to be fixed in the next release, when the Containerd support will be introduced.
/milestone v1.15
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
This is dependent of the ContainerD implementation. If that hasn't been implemented yet, this can't either.
Is there any update is the containerd implementation finished? Are there any relevant issues or PRs that we can use for tracking?
sounds like a no-go for 1.15, let's just move it out as no one is working on it as well. /milestone v1.16
Hello!

I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
/milestone v1.17 We won't be able to do this until containerd is ready. It might work in 1.17
Bug triage for 1.17 here. This issue has been open for a significant amount of time and since it is tagged for milestone 1.17, we want to let you know that the 1.17 code freeze is coming in less than one month on Nov. 14th. Will this issue be resolved before then?
We have a PR for it, as you can see above: It;s a pretty small PR too.
The v1.17 code freeze is this Thursday, end of day Pacific time. Will this issue be closed by then?
/milestone v1.18 (code freeze v1.17)
This is going to need a Windows fix, not a ContainerD fix. We won't have that before code freeze on 5 March, so I'm pushing this out to 1.19. /milestone v1.19
This is being tracked with MS internal work item The Windows team thinks an OS fix for Windows Server 2019 will be released with the June patches. I will post updates here as I get them.
Hello from the bug triage team! This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
can you comment on when we can expect a Windows fix for Windows Server 2019 that addresses the single file mount issues in containers when a target file already exists in the container?

we are waiting on a Windows OS fix from Microsoft to proceed here. If that fix comes in time for us to validate this functionality in 1.19 we would like it to be included in 1.19.
Hello! As this issue seems to depend on PR which has been pushed to v1.20, should this issue also be pushed to that milestone?
/milestone 1.20 The OS fix is now expect to be available as part of August Windows patches and then changes to the kubelet will need to follow. /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone.

<detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
yes it is expected be part of August Windows release
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
- Yes, we'd like to make it into 1.20. We were waiting on some bugfixes in the Windows OS and those bugfixes are now available (as of Oct 20 2020).
/remove-lifecycle frozen
/priority important-soon
I believe the work involved here is to update to enable this fucntionality for WIndows if container-runtime == remote (still disable this for DockerShim) Update/Add test coverage Update documentation - possible this section ?
/milestone v1.21 Moving to 1.21 since the required OS fixes were not available in time to implement this functionality in v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello! I am from the Bug Triage team! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
Can this be reopened? From a related problem I have experienced using HostAlias for Windows pod deployments, this issue does not seem to be resolved. Instead, it looks as if it was closed by accident because the conversation went cold.
Hello. What Kubernetes version are you running? Also, on the Windows nodes, do you use docker, or containerd?

This is only supported on Windows nodes with containerd, since it supports single file mapping, which is required in order to mount single files into Windows containers.
We have experimented with both 1.22.11 and a 1.23 release and yea, the container runtime is containerd. Any other ideas to check up on?


Question: podNamespace:          pod.Namespace, podServiceAccountName: pod.Spec.ServiceAccountName, volName:               spec.Name(), <ins> MetricsProvider:       metricsProvider, </ins> }, runner:   runner, spec:     spec,

kind feature What happened: Currently there is no mechanism to report metrics as the flex plugins are not enabled to collect metrics. All flex volumes lack this feature as compared to in-tree/built-in drivers even though it is possible to collect such metrics for most of the plugins. What you expected to happen: Depending on the availability of the metrics from the mounted FS, flex plugins should allow to report metrics for the FS. How to reproduce it (as minimally and precisely as possible): The default behavior that can be proved easily by just running a command similar to the following command on a node that is running a flex volume. This will show that there are no metrics collected for flex volumes. kubectl proxy $curl 127.0.0.1:8001/api/v1/nodes/<IP/proxy/stats/summary Environment**: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: local x86 OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux KubeStorage 4.4.0-130-generic -Ubuntu SMP Thu Jun 14 08:53:28 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: Others: /sig storage
+1 Yes, it will be good, if flex volume can support atleast the default metrics( i.e ), which most of the volume plugins support now. Based upon flex plugin "capabilities" parameters, it can be decided, whether the plugin supports "default metrics" or "nil metrics".


Question: // This is O(N), but we expect haystack to be small; // so small that we expect a linear search to be faster <del> func contains(haystack []int, needle int) bool { </del> <ins> func containsNumber(haystack []int, needle int) bool { </ins> for _, v := range haystack { if v == needle { return true

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --BUG REPORT /kind bug What happened: I tried to submit a Service Replace () to the kube-apiserver and this results the kube-apiserver container crashing and restarting. PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenameWhat you expected to happen: I expected the server not to crash and to get validation errors if the request was invalid. How to reproduce it (as minimally and precisely as possible): Create a service, get the resourceVersion and clusterIP from the created resource GET localhost:8001/api/v1/namespaces/<namespace/services/<servicenameSubmit a replace command to PUT localhost:8001/api/v1/namespaces/<namespace/services/<servicenamewith body like If the same request is submitted with the nodePort fields in the ports array then the request succeeds. I get that I may be sending an invalid request without a node port (though shouldn't the merge deal with that?) but it shouldn't take down the API server. Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T23:29:08Z", GoVersion:"go1.8.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:16:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Bare metal cluster OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-98-generic Install tools: Others:
thanks for reporting this! Seems like this is similar to and should be fixed by . can you PTAL that PR to see if it's good to merge?
/sig api-machinery
Duplicate of

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Re-applying a NodePort service that has two ports fails and crashes the API server: What you expected to happen: Do not crash, service is configured correctly. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: on VBox provider in macOS OS (e.g. from /etc/os-release): macOS/Minikube Kernel (e.g. ): Install tools: Others:
Seems that in func updateNodePorts, it ignored protocol when Detecting duplicate nodeport, I would have a fix on it :)
/sig network
FYI: I think this would be fixed in
/cc Maybe this is the time to have another review on and merge it :)
/assign
Fine, That makes sense

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /cc /assign What happened: a ClusterIP type service the service. Change it's type to NodePort and add a Port. Note: the new port has same port number but protocol with the existing Port. the updated service. The TCP Port was eaten. What you expected to happen: The updated service has both TCP and UDP ports. How to reproduce it (as minimally and precisely as possible): See the step above. Anything else we need to know?: If I create a NodePort service with the above two ports, it works. I write a test case to verify this case. I got a panic. The root cause is as below. We should take another look at this logic. BTW: changed some related code but it doesn't change the logic here. So I believe this bug is not introduced by . I also tested this on v1.5.1, it has the same issue. Environment: Kubernetes version (use ): HEAD Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I will be OOO the next week. Since this bug is not so urgent, I will fix it ASAP when I'm back. Folks who are interested in this can take it :) BTW, I can join in the discussion from my phone.
I'm not sure this is a bug. In the first configuration yaml, you're defining the ports array twice. I'm the second, you're only defining it once. The ports array should only be defined once.
Did you mean the below yaml by ? Please note that the two ports have same port number but different protocols. This is allowed.
Yes, that's the one I meant. But note you have "ports:" twice.
Ah, thanks for pointing it out! This might be the cause. But I remembered I also tested the case without the second line, it had the same issue. I can't verify this now. Will test it when I'm back to my keyboard.
I try the case without the second line, the issue also exists.

So I believe there is a bug here. Seems like there might be two issues here: a port is lost as is described in this issue. we shouldn't allow to configure two lines in the manifest. i.e. we should add validation for this case.
/remove-priority awaiting-more-evidence
can you please provide (or edit) example YAMLs then to allow for attempts at duplication?
Update: reproducing without the extra line. a ClusterIP type service the service (). Change it's type to NodePort and add a Port which has the same port number but different protocol with the existing Port. the updated service. We can see that the UDP Port was not created. I have dug into the code and will send a patch to fix it.
PR sent. Reviews welcome! Thanks!
There are/were bugs in the patch implementation wrt ports, because they need compound merge keys.
Yeah I had some discussios with in . But we also do have a bug in our service updating codes. See:


Question: Last updated 2016-11-08 08:09:17 UTC   Last updated 2016-11-18 14:07:18 UTC

API documentation says is rounded up, but it is actually rounded down: The code shows it is rounded down, not up: and
Yes, the documentation is obsolete. We changed maxUnavailable to round down so that we wouldn't accidentally infer 100% unavailability for non-100 percentages. For example, maxSurge=25% and maxUanavailable=25% with a Deployment of 1 replica meant that we could scale down 1 before thus leading in 100% maxUnavailable (implicit).


Question: errors = append(errors, fmt.Errorf("--max-resource-write-bytes can not be negative value")) } <ins> if err := validateHSTSDirectives(s.HSTSDirectives); err != nil { errors = append(errors, err) } </ins> return errors } <ins> func validateHSTSDirectives(hstsDirectives []string) error { // HSTS Headers format: Strict-Transport-Security:max-age=expireTime [;includeSubDomains] [;preload] // See https://tools.ietf.org/html/rfc6797#section-6.1 for more information allErrors := []error{} for _, hstsDirective := range hstsDirectives { if len(strings.TrimSpace(hstsDirective)) == 0 { allErrors = append(allErrors, fmt.Errorf("empty value in strict-transport-security-directives")) continue } if hstsDirective != "includeSubDomains" && hstsDirective != "preload" { maxAgeDirective := strings.Split(hstsDirective, "=") if len(maxAgeDirective) != 2 || maxAgeDirective[0] != "max-age" { allErrors = append(allErrors, fmt.Errorf("--strict-transport-security-directives invalid, allowed values: max-age=expireTime, includeSubDomains, preload. see https://tools.ietf.org/html/rfc6797#section-6.1 for more information")) } } } return errors.NewAggregate(allErrors) } </ins> // AddUniversalFlags adds flags for a specific APIServer to the specified FlagSet func (s *ServerRunOptions) AddUniversalFlags(fs *pflag.FlagSet) { // Note: the weird ""+ in below lines seems to be the only way to get gofmt to

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add the "Strict-Transport-Security" header to the API server responses. Why is this needed: Nessus scanner treats this as a Medium Risk in kubernetes 1.18.10.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig auth


Question: return volumes[0], nil } <ins> // applyUnSchedulableTaint applies a unschedulable taint to a node after verifying // if node has become unusable because of volumes getting stuck in attaching state. func (c *Cloud) applyUnSchedulableTaint(nodeName types.NodeName, reason string) { node, fetchErr := c.kubeClient.CoreV1().Nodes().Get(string(nodeName), metav1.GetOptions{}) if fetchErr != nil { glog.Errorf("Error fetching node %s with %v", nodeName, fetchErr) return } taint := &v1.Taint{ Key:    nodeWithImpairedVolumes, Value:  "true", Effect: v1.TaintEffectNoSchedule, } err := controller.AddOrUpdateTaintOnNode(c.kubeClient, string(nodeName), taint) if err != nil { glog.Errorf("Error applying taint to node %s with error %v", nodeName, err) return } c.eventRecorder.Eventf(node, v1.EventTypeWarning, volumeAttachmentStuck, reason) } </ins> // waitForAttachmentStatus polls until the attachment status is the expected value // On success, it returns the last attachment state. func (d *awsDisk) waitForAttachmentStatus(status string) (*ec2.VolumeAttachment, error) {

On AWS when we call "attach" of a volume then we periodically we check (via API call) if volume has indeed been attached. We run the check in a loop with timeout of 21 minutes. AWS documents that - if a volume is stuck in "attaching" state for too long, the node where volume is being attached is in "dirty" state and must be rebooted to fix the problem. Until node is rebooted, more pods with volumes to same node is only going be stuck in same state (i.e "attaching"). We propose that - if attach operation times out waiting for volume in "attaching" state, we should simply taint the node so as it can not affect other pods and a admin can manually fix the node. /sig aws /sig storage cc


Question: --overrides="": An inline JSON override for the generated object. If this is non-empty, it is used to override the generated object. Requires that the object supply a valid apiVersion field. --port=-1: The port that this container exposes. -r, --replicas=1: Number of replicas to create for this container. Default is 1. <del> -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile. </del> <ins> -t, --template="": Template string or path to template file to use when -o=template or -o=templatefile.  The template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview] </ins> Global Flags: --alsologtostderr=false: log to standard error as well as files

periodic-conformance-main-k8s-main Kubernetes e2e suite.SynchronizedBeforeSuite 07/17 23:52PDT We saw a similar issue in , which was fixed by upgrading the control plane OS from Ubuntu 18.04 to 22.04. /sig cluster-lifecycle
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
/sig node It seems to be related to SWAP.
There is a bug in containerd that is fixed in v1.6.14. The test here is using v1.6.8. Kubernetes v1.27 supported containerd versions: 1.7.0+, 1.6.15+
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still failing after is merged. Have we tried bumping the control plane OS image to ubuntu 22.04, like ?
Nevermind, the latest testgrid job might have been triggered before the PR was merged. I am observing this and will close this once I see the job passes.
Containerd version is already v1.6.21 in 's log There maybe still other problems.
can fix a similar failure in .
/close The failure of the swap is fixed by . The current failure is a flake that is mentioned in
Closing this issue. <detailsIn response to : /close The current failure is a flake that is mentioned in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ) type NodeAddress struct { <del> Type    NodeAddressType `json:"type"` Address string          `json:"address"` </del> <ins> Type    NodeAddressType `json:"type" description:"node address type, one of Hostname, ExternalIP or InternalIP"` Address string          `json:"address" description:"the node address"` </ins> } // ResourceName is the name identifying various resources in a ResourceList.

PR had a few errors in its API objects, where the "description" field errors in its JSON declarations were missing. But, the "verify-" presubmit check didn't flag them. This should be investigated and fixed.
Could you please take a look at this?


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package convert import ( "sort" "strings" "testing" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/client-go/kubernetes/scheme" "k8s.io/kubernetes/pkg/api/legacyscheme" ) func TestKnownVersions(t *testing.T) { legacytypes := legacyscheme.Scheme.AllKnownTypes() alreadyErroredGroups := map[string]bool{} var gvks []schema.GroupVersionKind for gvk := range scheme.Scheme.AllKnownTypes() { gvks = append(gvks, gvk) } sort.Slice(gvks, func(i, j int) bool { if isWatchEvent1, isWatchEvent2 := gvks[i].Kind == "WatchEvent", gvks[j].Kind == "WatchEvent"; isWatchEvent1 != isWatchEvent2 { return isWatchEvent2 } if isList1, isList2 := strings.HasSuffix(gvks[i].Kind, "List"), strings.HasSuffix(gvks[j].Kind, "List"); isList1 != isList2 { return isList2 } if isOptions1, isOptions2 := strings.HasSuffix(gvks[i].Kind, "Options"), strings.HasSuffix(gvks[j].Kind, "Options"); isOptions1 != isOptions2 { return isOptions2 } if isInternal1, isInternal2 := gvks[i].Group == runtime.APIVersionInternal, gvks[j].Group == runtime.APIVersionInternal; isInternal1 != isInternal2 { return isInternal2 } return gvks[i].String() < gvks[j].String() }) for _, gvk := range gvks { if alreadyErroredGroups[gvk.Group] { continue } if _, legacyregistered := legacytypes[gvk]; !legacyregistered { t.Errorf("%v is not registered in legacyscheme. Add group %q (all internal and external versions) to convert/import_known_versions.go", gvk, gvk.Group) alreadyErroredGroups[gvk.Group] = true } } } </ins>

What happened: Tried to convert a v1beta1 Ingress object to v1, and received the following error: What you expected to happen: The conversion to work. How to reproduce it (as minimally and precisely as possible): Then run: /sig cli /assign
Can i work on this issue?
/assign
/triage accepted


Question: # stop docker if it is present as we want to use just containerd log-wrap 'StopDocker' systemctl stop docker || echo "unable to stop docker" fi <del> log-wrap 'SetupContainerd' setup-containerd </del> <ins> if [[ ! -e "/etc/profile.d/containerd_env.sh" ]]; then log-wrap 'SetupContainerd' setup-containerd else echo "Skipping SetupContainerd step because containerd has already been setup by containerd's configure.sh script" fi </ins> log-start 'SetupKubePodLogReadersGroupDir' if [[ -n "${KUBE_POD_LOG_READERS_GROUP:-}" ]]; then

majority of tests Since 01-25 No response No response No response /sig node
Last passing - First failing -
Majority of tests failing with
Passing metadata Failing metadata
/cc
/assign
More investigation -- containerd version was bumped from passing run to failing run (from ) passing version: failing version:
I'm trying to compare the env setup to which is same except not on COS m97 which is passing fine. The failures in this version seem to be networking/connectivity related, lots of errors of connection not made and agnhost issues...
/triage accepted
Did some more digging on this run - I picked a single failure to investigate deeper - So one of the storage tests is attempting to exec a command on the node but the container is exited. The failure is here - To run the host exec commands it appears the test creates a . runs . The pod the test created was called . The audit logs have the pod spec for this pod - The pod appears to be scheduled on node. Here are the logs related to the pod from kubelet searching by pod name and UID It appears the pod starts successfully but then abruptly it is killed The restarts a few times and this pattern repeats. One thing stands out to me right before the container is killed, there is the following log saying that cpuset controller is missing...
I was able to repro the test failures locally now, with running it as follows: After investigating the cgroup error above, the issue became more clear -- the test is no longer configuring containerd correctly. Specifically, containerd is no longer configured to use the systemd cgroup driver. As background, the test is expected to pick up the containerd configuration from the containerd script and the systemd cgroup setting by using this . However, reproing the test, the containerd config is not picking up this change. Looking at bit deeper, it looks like the issue is that the containerd configuration is being overriden. Specifically, first the containerd setup scripts runs and writes the containerd config file () and then the k8s scripts runs and overwrites the config as part of calling . As a result any special config in the first containerd setup script ends up by overriden and lost. This is due to the changes introduced in the following two PRs: The current behavior of k8s setup scripts is now we always call .

Prior to , was only called if was set to : The related issue is that prior to , was set to . By setting to , the script did not call and the containerd config was preserved (since ). I think the larger issue is that prior to , there were two settings and for on . was used in jobs where containerd had to be installed manually and was used to indicate that the built in containerd version of the base OS will be used. Due to the combination of and , there is no longer distinction between and and the containerd config now is always being overridden by the k8s scripts.
Here is one way that it can be fixed: alternatively we can consider to revert the two PRs above but since they are quite large we might hit some conflicts, so it may be easier to rollforward with fix.
Last run is back to green, yay! <img width="1295" alt="image" src="https://user-"


Question: all: push # Set this to the *next* version to prevent accidentally overwriting the existing image. <del> TAG = 1.4 </del> <ins> TAG = 1.5 </ins> webserver: webserver.go CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -ldflags '-w' ./webserver.go

test panicked: (root)/Kubernetes%%20suite/Monitoringverifymonitoringpodsandallclusternodesareavailableoninfluxdbusingheapster/ cc
Vish please take a look if you have cycles or reassign if there's someone with better context.
I looked at the panic and it was due to the last test (networking) timing out but still running. I think the fix is more in better cleanup between tests.
I just saw this... it's totally due to ginkgo's cleanup when a test times out. I can fix.
I will get this one.
Ugh, the thing ginkgo does when a test times out is totally lame...
Instead of working around this, I've filed I'm going to go back to a manual timeout on this test.