Question: fhu := hostutil.NewFakeHostUtil(nil) fsp := &subpath.FakeSubpath{} <del> mounts, _, _ := makeMounts(&pod, "/pod", &container, "fakepodname", "", []string{""}, podVolumes, fhu, fsp, nil) </del> <ins> mounts, _, _ := makeMounts(&pod, "/pod", &container, "fakepodname", "", []string{""}, podVolumes, fhu, fsp, nil, false) </ins> expectedMounts := []kubecontainer.Mount{ {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig windows What happened: When deploying Kubernetes pods, the Linux containers will contain the following entries in the file: # Kubernetes-managed hosts file. 127.0.0.1 localhost ::1 localhost ip6-localhost ip6-loopback fe00::0 ip6-localnet fe00::0 ip6-mcastprefix fe00::1 ip6-allnodes fe00::2 ip6-allrouters 10.240.0.67 dns-querier-1.dns-test-service.e2e-tests-dns- dns-querier-1 However, these are not set on Windows containers; the container's file does not contain any entry. This can be attributed to the fact that individual files cannot be mounted on Windows containers the same way they are on Linux containers. It can be fixed with a change to the kubelet once Windows Server 2019 has shipped, and once CRI-ContainerD is enabled on Windows. We cannot fix it for Windows Server 1803 or earlier with dockershim. What you expected to happen: The file should be managed and populated by Kubelet. How to reproduce it (as minimally and precisely as possible): Spawn a Windows pod, and check its container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cmd /S /C type C:WindowsSystem32driversetchosts Compare the result with a Linux pod's container's file. kubectl create namespace e2e-tests-dns-x649p kubectl create -f # kubectl exec -n e2e-tests-dns-x649p dns-test--6d85-11e8-89cc- -c querier -- cat /etc/hosts Anything else we need to know?: Environment: Kubernetes version (use ): 1.12 Cloud provider or hardware configuration: N/A OS (e.g.

from /etc/os-release): Windows Server 1803 (10.0..48) Install tools: acs-engine v0.24.0 Others:
This requires single-file mapping, which won't work in Windows Server version 1803. It should be possible with Windows Server 2019 along with ContainerD
Will it work with Docker on Windows Server 2019?
Not in 18.09 or earlier. I'm not sure what release it will be in because it will require migrating to containerd.
should this issue be tracked for 1.13 in addition to If so please apply the appropriate priority and milestone labels please.
This one needs to be in v1.14. We need to get cri-containerd working first to fix this.
Thanks /milestone v1.14
By default, hosts file are not being used in Windows, it is all registry based. What features is this blocking? CC
The hosts file entries are still taken into account on Windows when it has to resolve a DNS name. This is blocking DNS conformance tests, like.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
Hello! 14 code freeze is coming in about 3 days, is this intended to be implemented in the following week?
No. It is meant to be fixed in the next release, when the Containerd support will be introduced.
/milestone v1.15
Hello! We are starting the code freeze for 1.15 tomorrow EOD. Is this issue still planned for the 1.15 cycle?
This is dependent of the ContainerD implementation. If that hasn't been implemented yet, this can't either.
Is there any update is the containerd implementation finished? Are there any relevant issues or PRs that we can use for tracking?
sounds like a no-go for 1.15, let's just move it out as no one is working on it as well. /milestone v1.16
Hello!

I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
/milestone v1.17 We won't be able to do this until containerd is ready. It might work in 1.17
Bug triage for 1.17 here. This issue has been open for a significant amount of time and since it is tagged for milestone 1.17, we want to let you know that the 1.17 code freeze is coming in less than one month on Nov. 14th. Will this issue be resolved before then?
We have a PR for it, as you can see above: It;s a pretty small PR too.
The v1.17 code freeze is this Thursday, end of day Pacific time. Will this issue be closed by then?
/milestone v1.18 (code freeze v1.17)
This is going to need a Windows fix, not a ContainerD fix. We won't have that before code freeze on 5 March, so I'm pushing this out to 1.19. /milestone v1.19
This is being tracked with MS internal work item The Windows team thinks an OS fix for Windows Server 2019 will be released with the June patches. I will post updates here as I get them.
Hello from the bug triage team! This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting June 25th (about 4 weeks from now) and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As the issue is tagged for 1.19, is it still planned for this release?
can you comment on when we can expect a Windows fix for Windows Server 2019 that addresses the single file mount issues in containers when a target file already exists in the container?

we are waiting on a Windows OS fix from Microsoft to proceed here. If that fix comes in time for us to validate this functionality in 1.19 we would like it to be included in 1.19.
Hello! As this issue seems to depend on PR which has been pushed to v1.20, should this issue also be pushed to that milestone?
/milestone 1.20 The OS fix is now expect to be available as part of August Windows patches and then changes to the kubelet will need to follow. /cc
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , , ] Use to clear the milestone.

<detailsIn response to : /milestone 1.20 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.20
yes it is expected be part of August Windows release
Hey Bug Triage here. Wanted to follow up on the status of this issue as we're approaching code freeze on 12.11.2020. This issue is tagged for 1.20, is it still planned for this release?
- Yes, we'd like to make it into 1.20. We were waiting on some bugfixes in the Windows OS and those bugfixes are now available (as of Oct 20 2020).
/remove-lifecycle frozen
/priority important-soon
I believe the work involved here is to update to enable this fucntionality for WIndows if container-runtime == remote (still disable this for DockerShim) Update/Add test coverage Update documentation - possible this section ?
/milestone v1.21 Moving to 1.21 since the required OS fixes were not available in time to implement this functionality in v1.20
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Hello! I am from the Bug Triage team! This issue has not been updated for a long time, so I'd like to check on the status. The code freeze is starting March 9th, 2021 (about 1 week from now). We want to ensure that each PR has a chance to be merged before code freeze. As the Issue is tagged for 1.21, is it still planned for this release?
Can this be reopened? From a related problem I have experienced using HostAlias for Windows pod deployments, this issue does not seem to be resolved. Instead, it looks as if it was closed by accident because the conversation went cold.
Hello. What Kubernetes version are you running? Also, on the Windows nodes, do you use docker, or containerd?

This is only supported on Windows nodes with containerd, since it supports single file mapping, which is required in order to mount single files into Windows containers.
We have experimented with both 1.22.11 and a 1.23 release and yea, the container runtime is containerd. Any other ideas to check up on?


Question: func (c *cacheWatcher) add(event *watchCacheEvent, budget *timeBudget) { // Try to send the event immediately, without blocking. select { <del> case c.input <- event: </del> <ins> case c.input <- *event: </ins> return default: }

Our scalability tests started failing on Thu last week: https://k8s- [250 run passed, 251 failed and since then everything failed] I started looking into it and it seems that something happened to how we are handling watches (i.e. I'm seeing a huge number of watches failing with "err too old" errors (which I didn't see before).
As an example in run 258 of kubemark-scale, I'm seeing this log in the test (after 5 minutes of running test); This is extremely strange and unexpected. I looked into apiserver logs, and there is expected number of POST and PUT operations on pods (and no DELETE and PATCH operations until then). Until then, there were alltogethere ~60.000 operations on pods. However, in 5000-node cluster, the size of watchcache of pods is supposed to be 250.000, which means that at that moment we should have all the history from the beginning in watch cache. And still we are observing those errors. Something must have changed around that recently...
As an example, logs from apiserver from a given node: This means something is really wrong: the first watch started at 45, then after it finished, we tried starting watch from rv= and it failed with "rv too old" the list, returned rv= from which we started the next watch. If the watch cache size for pods would be 250.000 (50*cluster size), there there is no way that with current rv=, the rv= would be outside history window...
I had a hypothesis that it might be the problem of breaking watch to etcd. But in that case there should be a warning "Terminating all watchers" log in apiserver, and there is no such log. It also seems that the size of cache is being set correctly...
So I tried running large kubemark locally and it seems that the problem is that the size of cache is too small. I few more logs, and here is what I'm seeing: So it seems that we are correctly initializing cachesizes, but then watch cache capacity is 100 (which explains why we are seeing all those "rv too old" errors. I will try to debug why the capacity is so small.
OK - I found it. contains the fix.


Question: return err } <del> klog.V(4).Infof("azureMount: mount options(%q) source:%q, target:%q, fstype:%q, begin to mount", </del> <ins> klog.V(4).Infof("mount options(%q) source:%q, target:%q, fstype:%q, begin to mount", </ins> options, source, target, fstype) <del> bindSource := "" </del> <ins> bindSource := source </ins> // tell it's going to mount azure disk or azure file according to options if bind, _, _ := isBind(options); bind {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: azure file remount on Windows in same node would fail What you expected to happen: If user delete a pod with azure file mount in deployment and it would probably schedule a pod on same node, azure file mount will fail since command would fail if file share is already mounted on the node: It's related to bug: Error logs would be like following: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.9-v1.12, master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /assign
/sig windows
/sig azure
/milestone v1.14 cc This may be good to backport to 1.12 or 1.13 too if customers are using those versions.


Question: } } <del> return field.ErrorList{} </del> <ins> return allErrs </ins> } // Validate container resource name

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In some validation functions, such as , , value returned may be wrong. When return some errors and , we may just return , that is not expected What you expected to happen: return correctly
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig api-machinery
/assign
Can you clarify "wrong"?
For example in will add and to field of ResourceQuotaSpec. Before , these two resources are not to standard resource sets, will return error since they are not fully-qualified neither. But will not return any error since the two resources are to standard quota resource name sets. The result above is not expected . Error happened in , we need to return it instead of returning when we ran


Question: Example: versionExample, Run: func(cmd *cobra.Command, args []string) { cmdutil.CheckErr(o.Complete(f, cmd)) <del> cmdutil.CheckErr(o.Validate()) </del> <ins> cmdutil.CheckErr(o.Validate(args)) </ins> cmdutil.CheckErr(o.Run()) }, }

ran instead of and got no error and then though that was the right version kubectl should fail for unexpected extra arguments
/sig cli
/assign
/triage accepted


Question: // MatchEndpoints returns a generic matcher for a given label and field selector. func MatchEndpoints(label labels.Selector, field fields.Selector) generic.Matcher { <del> return generic.MatcherFunc(func(obj runtime.Object) (bool, error) { endpoints, ok := obj.(*api.Endpoints) if !ok { return false, fmt.Errorf("not a endpoints") } fields := EndpointsToSelectableFields(endpoints) return label.Matches(labels.Set(endpoints.Labels)) && field.Matches(fields), nil }) </del> <ins> return &generic.SelectionPredicate{label, field, EndpointsAttributes} </ins> } <del> // EndpointsToSelectableFields returns a label set that represents the object // TODO: fields are not labels, and the validation rules for them do not apply. func EndpointsToSelectableFields(endpoints *api.Endpoints) labels.Set { return labels.Set{ "name": endpoints.Name, </del> <ins> // EndpointsAttributes returns the attributes of an endpoint such that a // generic.SelectionPredicate can match appropriately. func EndpointsAttributes(obj runtime.Object) (objLabels labels.Set, objFields fields.Set, err error) { endpoints, ok := obj.(*api.Endpoints) if !ok { return nil, nil, fmt.Errorf("invalid object type %#v", obj) </ins> } <ins> return endpoints.Labels, fields.Set{ "metadata.name": endpoints.Name, }, nil </ins> }

Using a watch expression on endpoints can lead to unpredictable behavior that includes returning a watch stream that ignores filters (e.g. a {name} determination) and returns items of the wrong type (e.g. changes to PODS in the ENDPOINTS stream). returns This result includes updates for services "kubernetes", "kubernetes-ro", "redis-slave", "redis-master", etc. Even though the requesting URL indicated only "guestbook" updates were to be watched. This seems to only occur when no resourceVersion is specified and the most recent resourceVersion has fallen off the etcd watch window. The expected behavior when resourceVersion is not specified I would think should be either: an initial update with the latest version of the resource, followed by filtered updates. nothing, followed by filtered updates. Though I'm not sure which of these is actually the spec. I would prefer (1) since it allows you to do a resource watch without having to first do a separate GET. But (2) coupled with a preceding GET would yield the same results.
IRC: cc
The pods mentioned in the stream are references to the pods corresponding to endpoints. See for discussion behind the feature. We probably need to distinguish the resourceVersion at which an object was last modified, which is also used as an update precondition, vs. the current (global) resourceVersion needed for watch.
Does the change to use GET with watch as a query parameter enable the GET to both return the latest version and subscribe to changes, or is a separate non-watch GET still required?
/subscribe
If you pass resourceVersion=0 you get list then watch (for single or multiple).
That behavior has been there for a while - the only challenge is you don't know when the list is complete, and we felt that list then watch more clearly delineated that.
This.
Watching with resourceVersion=0 should return an error, IMO.
I think the semantics for {watch a single resource} vs. {watch all resources of the same type} should be separated. When watching a single resource I would expect the watch to have consistent semantics. That is, I would expect it to be possible to watch an item and be guaranteed either: I saw every change to the item, or I knew that a change was missed (and so could do a raw GET to re-sync).

To this end it would be ideal if watch for a single item had the following semantics: Delivered the current state, followed by a list of all changes. If any change is missed or can't be delivered then the watch is terminated. Delivers either a guarantee of all changes that occurred on that resource from (exclusive) to present. If the server cannot prove that there were no missing changes then it should return an error and terminate the watch. This would imply that: Would always return an error unless the server can prove that it can deliver all changes to the resource (e.g. when the resource hasn't been created yet.) I'm surprised that the interface wants to provide multi-resource watch. It would seem that the semantics of this watch would either be potentially expensive to deliver or "best effort" such that it is basically useless for anything other than casual human consumption. Imagine a future where the master is sharded such that even different resources of the same type are on different shards. To provide meaningful semantics to a multi-resource watch would require that resource versions be meaningful in a global sense and in a way that spans all shards. Wouldn't this be undesirable? It would require that any write at any shard would need a round-trip message to all other shards to ensure that the logical clocks remain in sync. A multi-watch would similarly have to be fanned out to all shards in the backend. Both of these would defeat the benefits of sharding the data in the first place. I could understand providing a multi-resource watch for the creation and deletion of resources of a particular type (handled potentially by a secondary index in the sharded scheme) without providing a full watch of the contents. You could then use this watch to establish resource-specific watches for the resources you needed to track the contents of. Maintaining a secondary index wouldn't have to be transactional with changes that produced or destroyed items.
resourceVersion=0 is the same thing as saying watch resourceVersion=ObjectModifiedResourceVersion-1. Perhaps it should be more explicit, but it is generally useful. Why is blind watch bad? ----- Original Message -----
Also, I don't see it mentioned above but - everything in  is ignored. Confusing, but already fixed by the deprecation of .

cc
We don't offer multi-resource watch today, but in the future you're likely to correlate very strongly with a shard based on your namespace, rather than a shard on your type. I don't think sharding by type will be common except on a few types of resources (events, namespaces, possibly endpoints). The access pattern for most clients is going to be "in namespace Foo, get type X, get type Y, get type Z, get type A". Most UIs that want to watch anything will watch on the namespace. So the access pattern is less likely to reward sharding by type.
The query: Is a multi-resource watch. It watches for all resources of type Endpoints. But do you want the interface to guarantee that all objects in the same namespace are updated with consistency? I agree that shard boundaries are almost certainly NOT type-boundaries. For instance there is clearly a consistency relationship between a Service and its Endpoints object. I would hope that it would be impossible for me to observe a change to a Service object and then to subsequently query for its Endpoints object and get a stale read (semantically speaking). But that is a query across types. Suggesting that the appropriate shard boundaries are certainly not type-boundaries.
Sorry, I was referring to "multi-resource" as "watching over multiple resource types". It is possible to emulate a watch across a small number of stable shards by creating a resource version tuple (a vector clock, essentially) - for medium sized clusters I see that as a reasonable option. It's more likely though that infrastructure components that want to "watch the world" are either themselves shardable (such that it's acceptable for world to be defined as "my shard") or an intermediate component supports them. That intermediate component might be something like a partially ordered event stream that provides a firehose to the cluster as an aggregator of the cluster changes. Today, we require that API consumers assume resource version is internally consistent across a single type in a single namespace. Controllers are allowed to watch the world until such a time as we implement a sharding solution. There is no consistency relationship across resource types - services and endpoints are independent objects with independent lifecycles. It is by design that Endpoints can be updated out of band with Services, to allow downstream components to offload them.

We relax referential rules across resource types - there is no guarantee a referenced object will be created before or after you, or that it should exist only if you exist.
I completely agree there are possible implementations that can tolerate sharded and/or inconsistent queries. And deciding that certain kinds of JOINs, particularly across resource types and/or namespaces, are always going to be inconsistent seems like a fine place to land. I was merely surprised that the API was designed to promote the appearance of consistency for these queries today. I would have expected that the API would be more conservative to reserve these degrees of freedom around consistency for future implementations without requiring a future breaking change. Offering less functionality in V1 or functionality with weaker semantics would allow weakly consistent future implementations to emerge later without breaking existing applications. As a general rule I try to avoid APIs that give the appearance of consistency if they are not intended to actually deliver consistency. I think most developers will always assume consistency unless there is a really good reason to believe otherwise. This seems like human nature since people expect systems to exhibit the same causal behavior as the real world. Take for instance Endpoints. My impression of this after seeing the API was that Endpoints was simply a VIEW (in the SQL since) that provides a convenient way to express the JOIN between Service and Pods. From that perspective I would certainly have expected there to be a causal relationship between a change in the Service and its corresponding Endpoints object. I could easily imagine myself making that mistake when writing a application. Perhaps my program changes the Service object and then wants to send a message to all of the Endpoints associated with the new Service definition. After changing the Service, my program queries Endpoints. Did it get the new set of Endpoints? There are several possible outcomes: The Endpoints is the same as it was before I changed the Service Definition: The new Service definition just happens to result in the same set of Endpoints as before. I have not waited long enough for Endpoints to be updated. How long should I wait? What indicates that I've waited long enough? The Endpoints is different: Is this the correct one? The one that corresponds to my Service definition change? Or is this the update that occurred due to a different Service definition change that happened just before mine? If so, I should wait longer for mine.

Or is this the update that occurred due to a different Service definition change that happened just after mine? If so, I should either use this one or attempt my Service definition change again. In an inconsistent and asynchronously updated world we need to provide a way to answer these questions. (e.g. perhaps Endpoints should include the the resourceVersion of the Service definition used to compute it.) All applications need to properly deal with this extra information. (e.g. need to ignore all updated Endpoints until one arrives that has a resourceVersion = their change to the Service definition.) I'd imagine it's possible to implement this and for customers to use it correctly, but I think it offers a lot of opportunity for bugs. Users might simply wait for the next update and then take the wrong action leading to a misbehaving system. If we could decide which relationships should exhibit consistency and provide it where the developer would expect it then the API is simply and easier to use and applications are more likely to be correct. If we don't offer APIs which might lead the developer to make incorrect assumptions then applications are less likely to have those bugs.
I dug into this... it's more broken than I thought it was, so I've made a fix in . Once my fix goes in, a single object acts as if you didn't pass a name: this will be fixed directly (at least for the resource types I checked). get an appropriate resource version, you will have to LIST with a fieldselector ("?fields=name%3Dfoo" or some such-- don't recall exact syntax) instead of GETting. This way we don't have to modify the API to pass a special "watch at" resource version.


Question: ErrNodeUnschedulable = NewPredicateFailureError("NodeUnschedulable", "node(s) were unschedulable") // ErrNodeUnknownCondition is used for NodeUnknownCondition predicate error. ErrNodeUnknownCondition = NewPredicateFailureError("NodeUnknownCondition", "node(s) had unknown conditions") <del> // ErrTopologySpreadConstraintsNotMatch is used for EvenPodsSpread predicate error. ErrTopologySpreadConstraintsNotMatch = NewPredicateFailureError("EvenPodsSpreadNotMatch", "node(s) didn't match pod topology spread constraints") </del> // ErrFakePredicate is used for test only. The fake predicates returning false also returns error // as ErrFakePredicate. ErrFakePredicate = NewPredicateFailureError("FakePredicateError", "Nodes failed the fake predicate")

Part of /priority important-soon See example /sig scheduling /kind cleanup
/help
/assign
/remove-help


Question: RUN echo CACHEBUST>/dev/null && clean-install  iptables  <ins> e2fsprogs  </ins> ebtables  ethtool  ca-certificates

/kind bug What happened: When a cluster is provisioned using a containerised kubelet using the following images: Pods with PVCs are not created because kubelet cannot mount the disks. If you look at events, you can see that it's missing some binaries: This executable is . This problem can be resolved by mounting paths from the host into the rkt container, specifically and . Is this expected? mentioned in , that mounting operations are moving out from the main process into containers. Is this part of that effort? This caught me quite unexpected, and I didn't find any docs or changelog entries which indicated additional host mounts were necessary. If it's deliberate, I can close this until we perhaps document the breaking change. Is there a good place where we can document this? If it's not, I'll leave this open as a tracking issue. What you expected to happen: The hyperkube image to work without lots of host mounts. How to reproduce it (as minimally and precisely as possible): Run kubelet with rkt in one of the above listed images (see CoreOS' ). Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OpenStack OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: tectonic-installer
/sig storage /sig node
Seems the base hyperkube image was changed: done in /cc
The pkg/util/mount/mount_linux source defaults to ext4 if a fstype is not selected. It makes sense to restore the e2fsprogs package within the hyperkube image.
cc cc cc This seems like something that should be in the 1.8 milestone. Can someone who owns the hyperkube image comment on that?
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Containerization of mount is not going to happen until 1.9 at the earliest.

So if hyperkube is broken, it should be fixed before ship.
"modprobe" also went missing in 1.8 hyperkube images... Both kubelet and kube-proxy need it.
Are you noticing any buggy behaviour on those components without it? Would you mind creating an issue and/or PR to restore that dep
Done: And done: :)


Question: return w, nil }) <ins> // Create second pod which should preempt first pod. </ins> preemptorPod, err := createPausePod(testCtx.ClientSet, initPausePod(testCtx.ClientSet, &pausePodConfig{Name: "preemptor-pod", Namespace: testCtx.NS.Name, Priority: &highPriority, Resources: &resourceRequest})) if err != nil {

Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestCoSchedulingWithPermitPlugin Testgrid link: Reason for failure: /sig scheduling
Could be related
/help
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another run
Hmmm I think test plugin is no longer correct: After my change is executed at the end of the scheduling cycle, this means there can be only one happening at any given moment. If there is a race where "signaling-pod" is scheduled before "waiting-pod", then "signaling-pod" will wait 30s in in for the "waiting-pod" to become waiting, but this cannot happen. Basically "signaling-pod" is allowing "waiting-pod" before "waiting-pod" is being scheduled. I think test and test plugin could be changed, so there are no longer two pods of different kind. Instead first pod that arrives to the is considered "waiting-pod" and second pod is always "signaling-pod" that will either reject or allow first pod. This way we do not have to care about order in which scheduler will process these pods. I think it is also consistent with how actual co-scheduling will work.
Can you first confirm that this is what is happening? you could increase the log level to 3 to see the log lines in and . I feel like if that's the case, we have a separate bug with the event handlers or sorting. Other than that, I agree that the implementation of the test's shouldn't have waits. cc
My hope would be that the pod is temporarily unschedulable due to nodes still being synced.
Where you able to confirm the reason of the flake?
not yet, what is the easiest way to run this test multiple times? Should I try to reproduce this locally or create some temporary PR just for this purpose?

I wrote PR for the test fix to remove sleep from the PermitPlugin, it had to be done sooner than later regardless of the cause of flakiness.
Understanding the reason of the flake is important to make sure we are fixing the right thing and that we won't have future flakes. Please try to run the tests locally about 10 times.
It should be something like: If that doesn't work, run first
Flake point:
From the logs provided by you can tell that the problem is a race between "signalling-pod" and "waiting-pod" and waiting inside the Permit(), as I mentioned in this issue. BLUF: In the first case of the TestCoSchedulingWithPermitPlugin "waiting-pod" and "signalling-pod" pods are being retried many times, most of the time "signalling-pod" is rejecting already waiting "waiting-pod", but sometimes race can happen and "signalling-pod" will be processed by the scheduler before rejected "waiting-pod" (there is no guarantees whether rejecting pod or rejected pod will end up in the queue first). When this happens "signalling-pod" waits 30s inside the Permit() for "waiting-pod" to become waiting and blocks the scheduler, meanwhile second test case starts, "signalling-pod" eventually stops waiting for "waiting-pod" and gets bound, after that "waiting-pod" cannot be allowed, which leads to a failure of the second test case. My PR should fix this issue.
I feel like we should avoid sharing the scheduler & permitPlugin instances among the 2 subtests, which can then solve the flake?
I think this flake (in different variations) can happen as long as test depends on the order in which signalling and waiting pod enter the permit plugin.
I believe the PR might fix the issue. But since the reason for the flake is that one subtest is corrupting the other, it's possible that we might surface other flakes. That's why it's important that you run the test several times to check for other flakes. It several tests pass, let's proceed with this PR and leave a TODO to remove the dependency between the subtests.
I ran 10+ times on my branch with the fix and had no issues.


Question: }, }, { <del> name: "existing pods with mis-matched namespace doesn't count", </del> <ins> name: "existing pods in a different namespace do not count", </ins> pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(),

How to reproduce it (as minimally and precisely as possible): pre-requisites: a multi-az cluster, a deployment with 1 pod in each az. a with and . the node(s) in 1 az. the deployment's pod(s) from the cordoned az. What happened: A new pod was created and scheduled in a form that What you expected to happen: I expected that a new pod will be created and be "stuck" in status as it shouldn't have nowhere be scheduled. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ): 4.19.84-coreos Others: I created a with how I reproduce the bug
/sig scheduling
cc/
Thanks for raising this issue. I feel like it's a general timing issue - when you delete a Pod, that Pod may still exist for a very short time (in graceful deletion period), and at the same time, the controller backfills a Pod for it. So when scheduling tries to schedule the backfill Pod, it may read the cluster as: [1, 1, 1(deleting)]. To confirm this, can you try deleting the Pod using , and see if the problem goes away?
Yes sure I will try that. It sounds reasonable, thats why in the first demonstration I deleted all the pods and waited for all of them to completely disappear. Nevertheless, this is still something that I believe needs to be fixed. The pod's evaluation should take in consideration pods that are in status and don't count them as pods.
I do believe we exclude the Pods. (if not, it should be a serious regression). What concerns me is that at that moment, scheduler may treats that (deleting) Pod as running in its cache. I will try to debug into this.
Yep, this time the new pod was in status until I uncordoned the relevant nodes.


Question: UnavailableReplicas: 0, }, <del> msg:  "Waiting for rollout to finish: 0 out of 1 new replicas have been updated...n", </del> <ins> msg:  "Waiting for deployment "foo" rollout to finish: 0 out of 1 new replicas have been updated...n", </ins> done: false, }, {

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: In our CI, we run multiple times in parallel when we update multiple deployments. This lets us see the status of all deployments as they run. When a deployment succeeds, the final message tells us the name of the deployment. However, the intermediate "Waiting for rollout to finish" messages do not say the name of the deployment. This means that in order to look at multiple rollouts in parallel, we need to manually pipe through another command to prefix it with the rollout name. What you expected to happen: Each line to mention the rollout name. How to reproduce it (as minimally and precisely as possible): Update any deployment, run , note that intermediate status names don't contain the deployment name. Anything else we need to know?: Arguably another fix would be to allow you to pass multiple rollouts to and have kubectl handle polling multiple deployments in parallel. Environment: This is still true on master by code inspection.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } func prepareSnapshotDataSourceForProvisioning( <del> node e2epod.NodeSelection, </del> <ins> f *framework.Framework, config volume.TestConfig, </ins> client clientset.Interface, dynamicClient dynamic.Interface, initClaim *v1.PersistentVolumeClaim, class *storagev1.StorageClass, snapshotClass *unstructured.Unstructured, <ins> mode v1.PersistentVolumeMode, injectContent string, </ins> ) (*v1.TypedLocalObjectReference, func()) { var err error if class != nil {

What happened: https://k8s- These tests failed consistently: Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /assign /sig storage
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: https://k8s- Anything else we need to know?: Volume snapshot tests need to be reworked to use / helper functions that support block volume mode. In addition, csi-driver-host-path fails when restoring block volume snapshots. /sig stroage Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hostpath driver fix:
E2e test fix: (needs hostpath driver release first).


Question: "crypto/x509" "encoding/json" "fmt" <del> "io" </del> "io/ioutil" "net" "net/http" "net/http/httptest" "net/http/httptrace" <del> "os" </del> "reflect" "strings" "sync"

/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?


Question: func findRule(rules []network.LoadBalancingRule, rule network.LoadBalancingRule) bool { for _, existingRule := range rules { <del> if strings.EqualFold(*existingRule.Name, *rule.Name) && </del> <ins> if strings.EqualFold(to.String(existingRule.Name), to.String(rule.Name)) && </ins> equalLoadBalancingRulePropertiesFormat(existingRule.LoadBalancingRulePropertiesFormat, rule.LoadBalancingRulePropertiesFormat) { return true }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: controller manager crash looped with a seg fault after failing to create a load balancer for the kubernetes dashboard. The calls to create the load balancer were failing with a 400 What you expected to happen: controller manager to provision the resources or to fail without a segfault and a clearer error message saying what the error creating the resource was. How to reproduce it (as minimally and precisely as possible): Just created a cluster with an add on for the dashboard. Anything else we need to know?: logs from controller manager: I0423 12:50:28. ] Starting range CIDR allocator I0423 12:50:28. ] Waiting for caches to sync for cidrallocator controller I0423 12:50:28. ] Starting NoExecuteTaintManager I0423 12:50:28. ] Initializing eviction metric for zone: westeurope: :0 W0423 12:50:28. ] Missing timestamp for Node aks-nodepool1--0. Assuming now as a timestamp. I0423 12:50:28. ] Controller detected that zone westeurope: :0 is now in state Normal. I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Node", Namespace:"", Name:"aks-nodepool1--0", UID:"-4476-11e8-9b1c-", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'RegisteredNode' Node aks-nodepool1--0 event: Registered Node aks-nodepool1--0 in Controller I0423 12:50:28. ] Caches are synced for ClusterRoleAggregator controller I0423 12:50:28. ] Caches are synced for endpoint controller I0423 12:50:28. ] Caches are synced for attach detach controller E0423 12:50:28. ]

Error creating spec for volume "mssqldb", pod "feature-test3"/"mssqlserver--r5r7l": error processing PVC "feature-test3"/"mssql-data": PVC feature-test3/mssql-data has non-bound phase ("Pending") or empty pvc.Spec.VolumeName ("") I0423 12:50:28. ] Caches are synced for deployment controller I0423 12:50:28. ] Caches are synced for persistent volume controller I0423 12:50:28. ] Caches are synced for ReplicationController controller I0423 12:50:28. ] Caches are synced for TTL controller I0423 12:50:28. ] Caches are synced for cidrallocator controller E0423 12:50:28. ] Failed to process service kube-system/kubernetes-dashboard. Retrying in 5s: error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"kubernetes-dashboard", UID:"-4476-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"193", FieldPath:""}): type: 'Warning' reason: 'CreatingLoadBalancerFailed' Error creating load balancer (will retry): error getting LB for service kube-system/kubernetes-dashboard: Service(kube-system/kubernetes-dashboard) - Loadbalancer not found I0423 12:50:28. ] Event(v1.ObjectReference{Kind:"Service", Namespace:"feature-initialproject", Name:"identityserver-service", UID:"-4485-11e8-9b1c-", APIVersion:"v1", ResourceVersion:"7879", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer I0423 12:50:28. ] Caches are synced for namespace controller I0423 12:50:28. ] Caches are synced for service account controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ] Caches are synced for resource quota controller I0423 12:50:28. ]

Caches are synced for garbage collector controller I0423 12:50:28. ] Garbage collector: all resource monitors have synced. Proceeding to collect garbage I0423 12:50:28. ] Caches are synced for garbage collector controller I0423 12:50:28. ] Caches are synced for stateful set controller I0423 12:50:28. ] Caches are synced for disruption controller I0423 12:50:28. ] Sending events to api server. I0423 12:50:29. ] selectLoadBalancer: cluster(kubernetes) service(feature-initialproject/identityserver-service) isInternal(true) - availabilitysetsnames [nodepool1-availabilitySet-] E0423 12:50:30. ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) /usr/local/go/src/runtime/asmamd64.s:509 /usr/local/go/src/runtime/asmamd64.s:2337 panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1d5772d] goroutine 1338 [running]: (0x0, 0x0, 0x0) +0x111 panic(0x40b2e00, 0xa603590) +0x283 (Cloud).getServiceLoadBalancerStatus(0xc4206cdc00, 0xc422b401e0, 0xc4208f6b90, 0xc422b401e0, 0xc4210c0388, 0x1) +0x30d (Cloud).EnsureLoadBalancer(0xc4206cdc00, 0x496be8b, 0xa, 0xc422b401e0, 0xc4210c0388, 0x1, 0x1, 0xbeaf93852d9a680b, 0x2fe0fa757, 0xa9f0d40) +0x2ea (ServiceController).ensureLoadBalancer(0xc420f56000,

0xc422b401e0, 0xc422b401e0, 0x4961da9, 0x6) +0xcc (ServiceController).createLoadBalancerIfNeeded(0xc420f56000, 0xc421fd81e0, 0x2d, 0xc422b401e0, 0xc422c8dc40, 0xc422c8dc78, 0x1265492) +0x20e (ServiceController).processServiceUpdate(0xc420f56000, 0xc4217bcc60, 0xc422b401e0, 0xc421fd81e0, 0x2d, 0x0, 0x0, 0x0) +0xeb (ServiceController).syncService(0xc420f56000, 0xc421fd81e0, 0x2d, 0x0, 0x0) +0x3aa (ServiceController).worker.func1(0xc420f56000) +0xd9 (ServiceController).worker(0xc420f56000) +0x2b (ServiceController).()-fm() +0x2a (0xc420a04570) +0x5e (0xc420a04570, 0x3b9aca00, 0x0, 0x1, 0xc4208fa0c0) +0xbd (0xc420a04570, 0x3b9aca00, 0xc4208fa0c0) +0x4d created by (ServiceController).Run +0x20c Environment: Kubernetes version (use ): 1.9.1 Cloud provider or hardware configuration: Azure, AKS OS (e.g. from /etc/os-release): Ubunutu Kernel (e.g. ): xenial Install tools:AKS Others:
not sure what sig should be labeled in on this. Seems like a bug in sig-azure and whoever owns the piece getting the nil pointer/panic
what's the k8s version?
1.9.1
- I suspect that the error is here. This can be either one of two things A bug in SDK serialization (LB does have a private ip) but it does not get serialized correctly.

A bug in our assumption that that each internal LB ip fe config will have a valid internal IP. Can you dump the model for this internal LB here? Also -- did this error eventually recover? (it could be a race condition between setting ip and getting ip)
this was actually a customer cluster and they deleted it before I got a chance to get things like the model/what error code on the 400 was. And it didn't ever recover.
according to the evente table it was an internal loadbalancer here is the link to it --https://jarvis- maybe it the Details logged Can help you further
I was able to get more details from logs, {"error":{"code":"LoadBalancerFrontendIPConfigurationCountLimitReached","message":"Basic sku load balancer /subscriptions/redacted/resourceGroups/redacted/providers/Microsoft.Network/loadBalancers/kubernetes-internal cannot have more than 10 FrontendIPConfigurations.","details":[]}}
Thanks - So there are two errors here. One the user can not use more than 10 IPs. The fact that we panic on this. i believe the fix is to log and fail the operation. /assign
The issue has been fixed in and has been included in v1.9.2. Could you upgrade your cluster (e.g. to latest v1.9.7) and check whether it fixes the issue?
I'm closing this issue since it's been resolved. /close
I am seeing this on 1.10.x and 1.11.x clusters: 10.6 11.2
The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that /reopen
Reopening this issue. <detailsIn response to : The PR has already been included in v1.11.x and v1.10.x. However, there's still other potential issues. Let me reopen this issue and file another PR to fix that Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Added this to v1.12 since this is a critical urgent bug.


Question: force := false timeout := time.Duration(0) <ins> wait := false </ins> filenames := []string{} recursive := false

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc


Question: } zones := testutil.GetZones(fakeNodeHandler) for _, zone := range zones { <ins> // Time for rate-limiter reloading per node. time.Sleep(50 * time.Millisecond) </ins> nodeController.zonePodEvictor[zone].Try(func(value TimedValue) (bool, time.Duration) { uid, _ := value.UID.(string) deletePods(fakeNodeHandler, nodeController.recorder, value.Value, uid, nodeController.daemonSetStore)

Running latest from master (currently commit ), I'm seeing TestMonitorNodeStatusEvictPodsWithDisruption is flakey: test -c ./pkg/controller/node true; do if ! ; then break; fi; done for failure I've narrowed it down , which is supposed to process all the items (nodes) in the queue. This queue has a capacity of 1, and at least in the test code, a refill rate of QPS. If the call to fails , it will stop processing wherever it happened to be in the queue. I've some extra logging locally, and when it fails, I see it processing nodes 1 and 2 before TryAccept fails. The issue is that the pod it's trying to evict is on node0, and it doesn't process node0 before TryAccept returns false. cc
I'm surprised we haven't hit this in jenkins yet
and removed the "leak" option referred to in the godoc, but it seems like it kept the leak behavior, not the retry behavior when the rate limiter is hit
Still seeing this issue
I didn't understand how this is related to sig/scheduling. Am I missing anything here?
guessing the flake is related to this line: anything on the order of milliseconds is going to be unreliable in testenvs that sometimes peg the CPU with parallel tests or it's related to the single item burst and the "abort when TryAccept fails" behavior
I can extend this sleep to something bigger, but I don't have cycles to debug it properly:/
any thoughts on that?
Generally we retry when TryAccept fails, but not in tests. So either we need to rewrite the test, or lengthen the sleep time. It's not perfect, but there's no way for "wait until rate limiter is reloaded".
I was working on this issue and I tried the change that you suggested and I notice that it fails with this change too() and I think rewriting the test with retry seems to be the option as I tried increasing 5000 ms and still it failed. I am new to the codebase, if there is piece of code where retry is available, I can check on that.
- what you wrote is concerning. I wish I had time to debug it:/ Frankly - those tests are old. Like really, really old. So they start to deserve a rewrite.

On the flip side - we'll need to rewrite them when making taint-based evictions default, so we'll need to allocate time for that anyway. If lengthening sleep time reduce the flakyness I vote for doing that, and just wait for upcoming rewrite.
having a delay in the loop before caused it to run for close 4 hours without terminating on 8GB,4 core machine. Is this something normal? I know, is not happy with this change but since we are rewriting this test anyways, this could be a work around that removes the flakyness. Let me know your thoughts. If this does work, can we remove the flake label and keep this issues as this is not a release blocker for 1.6?
- I don't understand which delay in which loop you mean. You a delay inside the "Try" call? What do you mean by 'run for close 4 hours without terminating'? That the test didn't fail? If so, then I'm all for such solution for stopping the bleeding.
, Yes I introduced a sleep in loop for zone, like Yes the test didn't fail but it didn't complete either even after 4 hours, so I am wondering if this causes timeout in testing infra.
It's strange indeed... Do you have any logs for the test by any chance? Adding this delay shouldn't be noticeable, as it just inserts some delay here: Sleeping under the lock is usually not the best idea, but 50 microseconds in test shouldn't make a lot of difference...
Unfortunately no, I can try reproducing this today. Probably by afternoon EST, I can have something, The way I am doing as of now is the way suggested by in the comments above. I can redirect that to a file and see. Let me know, if you are ok with approach.
I'm OK with anything that produces logs:) I generally do:
- out of curiosity. How often does this flake?
If I run my loop, it takes about 8-10 seconds to flake. , Marek Grabowski <:
I think I made a mistake. I did not take that while true loop out because of which it ran in infinite loop. I am sorry about that.

It means that with that delay introduced that I mentioned earlier it never failed even after running for 4 hours, which means we got rid of atleast test flakyness. if you can confirm if it is working, I will raise a PR which introduces this delay.
giving the ratelimiter time to refill inside the loop is consistent with the theory that is causing problems. If the rate limiter runs out of tokens in Try(), it returns with a comment, but the unit test does not try again later. I'm having a hard time following the code to know if non-test invocations have the same problem or if this is just a unit test issue.
The sleep addition seems to have helped. rgudimet <:
It's test only problem: Runs in , so it'll retry in the next round. I guess the problem here was actually introduces later than what suggests - I don't think that the removal of leaky behavior was problematic, but introducing a test that evicts more than one Node was. This means that if processing queue is fast enough, then rate limiter fails to reload, so adding the sleep there is actually a correct solution (as long as we don't have good rate limiter mock/testable rate limiter).
Thanks for confirmation. I will raise a PR for that now. As far as I understood, the issue is with unit test. If rate limiter fails to reload for a particular node, it cannot retry(unless the we separate out the logic which tells if the rate limiter has reloaded or not. - if we have that logic we can ask Try to run again by returning a false just like ).
Still an issue: https://k8s-
but not a blocker (the test is at fault)
Thanks, I will check this out after 1.6 release hopefully, with rewrites..
Has there been any recent incidents of this issue? It's marked for 1.7.
Closing - shows near nothing.
- this flakes out sporadically, I was able to reproduce this when debugging on my system, if I run it for long time.
https://k8s-


Question: func init() { api.Scheme.AddDefaultingFuncs( <ins> func(obj *ReplicationController) { var labels map[string]string if obj.Spec.Template != nil { labels = obj.Spec.Template.Labels } // TODO: support templates defined elsewhere when we support them in the API if labels != nil { if len(obj.Spec.Selector) == 0 { obj.Spec.Selector = labels } if len(obj.Labels) == 0 { obj.Labels = labels } } }, </ins> func(obj *Volume) { if util.AllPtrFieldsNil(&obj.VolumeSource) { obj.VolumeSource = VolumeSource{

It doesn't make sense to define pod template with a set of labels that is strict subset of replicaSelector (or at least I can't find an use-case). For that reason I think that the labels in pod template part of replication controller should be optional and user should care about them only if he wants to define a superset of what replicaSelector already defines.
I've heard the same comments from a client. I think that it'd be more intuitive if we always specify pod labels in the template (so that we can copy&paste pod definition), and replicaSelector is optional (by default pod labels).
It would be reasonable for the selector to be a subset. I'm fine with intelligent defaulting but we need to be careful not to conflate the 2 concepts, which I think would increase the amount of confusion.
Setting RC selector from pod template labels SGTM. Any concerns It wouldn't have to work with separate templates, though it would be desirable if it did. It would be nice if we could add preconditions based on other objects (resource version of pod template in this case) to etcd transactions.
I think we agreed in another issue that defaulting the selector from the pod template labels is reasonable. No argument from me. One problem with pod templates is that we lose the strict relationship between selector and template, so if you change the template the RC needs to behave intelligently (I refuse to create more pods because the template selector doesn't match mine, etc)
We should do this, and we should default replicas to 1. We could also consider defaulting replication controller labels from pod template labels. That way, essentially the only thing that would need to be specified for a singleton RC would be the pod template.
I don't think we can default replicas to 1, since we can't tell the difference between an explicit zero and a default zero, and I think there are lots of reasonable cases where you want to set replicas == 0 on create. (esp. in the context of socket activation)
Responded on the PR: We can distinguish default from 0 if we make it a pointer.


Question: platforms  = flag.String("platform", "", "comma-separated list of platforms to typecheck") timings    = flag.Bool("time", false, "output times taken for each phase") defuses    = flag.Bool("defuse", false, "output defs/uses") <del> serial     = flag.Bool("serial", false, "don't type check platforms in parallel") </del> <ins> serial     = flag.Bool("serial", false, "don't type check platforms in parallel (equivalent to --parallel=1)") parallel   = flag.Int("parallel", 4, "limits how many platforms can be checked in parallel. 0 means no limit.") </ins> skipTest   = flag.Bool("skip-test", false, "don't type check test code") tags       = flag.String("tags", "", "comma-separated list of build tags to apply in addition to go's defaults") ignoreDirs = flag.String("ignore-dirs", "", "comma-separated list of directories to ignore in addition to the default hardcoded list including staging, vendor, and hidden dirs")

Background: We recently introduced Guaranteed Pod QOS on jobs running in GCP project (). Introducing limits on jobs naturally started causing some failures, one of which was (). The original memory limit of caused the pod to be unscheduleable, so it was bumped down to () at which point the pod became scheduleable and the job ran successfully (). At that point, after observing metrics, I mistakenly concluded that this was much higher than needed and suggested bumping down again (). This caused the job to continue to run, but to fail on and sometimes . was recently refactored to not depend on GOPATH (). This was a fairly significant change so I looked at the metrics before and after the rewrite was merged. (June 25th) ! (June 27th) ! Memory usage nearly double for post-merge, which appears to be directly related to the rewrite given that on other branches remained relatively constant. This can be affirmed further by the fact that and are failing with the new resource limits, while older release branches are not. I can see a few options for how we may be able to remedy this issue in the short term: Revert the rewrite to get back to lower (yet still very high) memory usage Separate into a separate job to not reduce signal on the rest of Bump memory limits back up to to get this running successfully again for now I plan to look into fixing up but think it would be a good idea to take one of the directions mentioned above if this cannot be remedied quickly. /priority critical-urgent /sig testing /cc
latest run is green
hmm


Question: framework.DeletePodWithWait(f, f.ClientSet, pod) }() <del> err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, 30*time.Second) </del> <ins> err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, framework.PodStartShortTimeout) </ins> Expect(err).To(HaveOccurred(), "while waiting for pod to be running") selector := fields.Set{

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: Looks like the newly tests are failing: https://k8s-
/assign Also random question to why is the same test run 4 times?
GitHub didn't allow me to assign the following users: kevtaylor. Note that only and repo collaborators can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
can anyone shed some light on why events are not being returned from a pod in the alpha suite? This has a 1 minute pod timeout and everything works fine locally
Reiterating the mentions to trigger a notification: <detailsIn response to : can anyone shed some light on why events are not being returned from a pod in the alpha suite? This has a 1 minute pod timeout and everything works fine locally Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I suggest unless anyone can define why the events do not appear, we disable that part of the test and just check for error. If you are okay with that please reopen the bug and I'll put in a new fix
it looks like node alpha suite is having some other issues right now that are preventing the tests from being run. So it's not clear right now if the fix helped or not.
it failed again. I did a comparison of a locally running one and the alpha suite. In the log file https://k8s- there are these references But in the locally running tests I don't see that step This running And the events running locally
/reopen


Question: } } <del> func TestApplyMultipleObject(t *testing.T) { </del> <ins> func TestApplyMultipleObjectsAsList(t *testing.T) { testApplyMultipleObjects(t, true) } func TestApplyMultipleObjectsAsFiles(t *testing.T) { testApplyMultipleObjects(t, false) } func testApplyMultipleObjects(t *testing.T, asList bool) { </ins> nameRC, currentRC := readAndAnnotateReplicationController(t, filenameRC) pathRC := "/namespaces/test/replicationcontrollers/" + nameRC

kubectl create, delete, etc. can already take a List object. kubectl apply can take the list object when it does creation, but it gives an (ugly) error when it's configuring the objects. Then... expected: Actual:
what's you ?
0-alpha.5
Sounds like a bug. Are you interested in taking a look
sure I will take a look during this weekend.
I think the root cause is , the actually is , in the origin , is not , but after the , this object is nil. Would you like to double check this?
I'll take a look.
if you have make sure about this, please let me know. Thanks :)
Is there any ways we could get this into 1.2? It's actually making our config generation quite painful to have to make a separate file for every object. It looks like got pretty far? would you like to make a PR that we can review?
I don't see how. It could have made 1.2 if a PR had been sent within a few days of the issue being filed.
I unassigned all issues assigned to you. Please only assign issues that are actively being worked on, so that others can know which issues are available to be picked up.
My apologies. If I'm not mistaken, this issue was filed after the code freeze for 1.2. I didn't realize that it was time critical or that it could make the release at that time. However, I should have asked. Sorry for the inconvenience. Thank you. I still plan to address those issues for the next release, as discussed, but will assign each one to myself as I work on it.
I created which fixes this. I'm really hoping we can get this into 1.2 since it's a pretty small, isolated fix but a pretty big bug that is complicating our config generation system. Let me know what else I can do to help.
Thanks for the PR. We're short on reviewer bandwidth at the moment. I assigned it to Clayton for review or delegation.


Question: } } <ins> func Test_syncEndpoint_updateWeightsOnRestart(t *testing.T) { tcpProtocol := v1.ProtocolTCP ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) fp := NewFakeProxier(ipt, ipvs, ipset, nil, nil, v1.IPv4Protocol) svc1 := makeTestService("ns1", "svc1", func(svc *v1.Service) { svc.Spec.ClusterIP = "10.20.30.41" svc.Spec.Ports = []v1.ServicePort{{ Name:     "p80", Port:     int32(80), Protocol: v1.ProtocolTCP, }} }) epSlice1 := makeTestEndpointSlice("ns1", "svc1", 1, func(eps *discovery.EndpointSlice) { eps.AddressType = discovery.AddressTypeIPv4 eps.Endpoints = []discovery.Endpoint{{ Addresses: []string{"10.180.0.1"}, }} eps.Ports = []discovery.EndpointPort{{ Name:     pointer.StringPtr("p80"), Port:     pointer.Int32(80), Protocol: &tcpProtocol, }} }) // sync proxy rules to get to the desired initial state makeServiceMap(fp, svc1) makeEndpointSliceMap(fp, epSlice1) fp.syncProxyRules() serv := &utilipvs.VirtualServer{ Address:   netutils.ParseIPSloppy("10.20.30.41"), Port:      uint16(80), Protocol:  string(tcpProtocol), Scheduler: fp.ipvsScheduler, } vs, err := fp.ipvs.GetVirtualServer(serv) if err != nil { t.Errorf("failed to get virtual server, err: %v", err) } rss, err := fp.ipvs.GetRealServers(vs) if err != nil { t.Errorf("failed to get real servers, err: %v", err) } for _, rs := range rss { rs.Weight = 0 if err = fp.ipvs.UpdateRealServer(vs, rs); err != nil { t.Errorf("failed to update real server: %v, err: %v", rs, err) } } // simulate a restart by enabling initial sync logic. fp.initialSync = true err = fp.syncEndpoint(proxy.ServicePortName{ NamespacedName: types.NamespacedName{ Name:      "svc1", Namespace: "ns1", }, Port:     "80", Protocol: tcpProtocol, }, true, vs) if err != nil { t.Errorf("failed to sync endpoint, err: %v", err) } rss, err = fp.ipvs.GetRealServers(vs) if err != nil { t.Errorf("failed to get real server, err: %v", err) } for _, rs := range rss { if rs.Weight != 1 { t.Logf("unexpected realserver weight: %d, expected weight: 1", rs.Weight) t.Errorf("unexpected realserver state") } } } </ins> func TestIPv4Proxier(t *testing.T) { tcpProtocol := v1.ProtocolTCP tests := []struct {

Entry in ipvs real sever remains at weight=0 after kube-proxy restarts. (During 1.21 -1.22 kubernetes cluster upgrade.) Related to As the related issue is considered closed, I will report this issue again as the problem still persists in newer version. Might because of some corner cases. Real server weight should be 1 when endpoints is ready. Use kube-proxy in ipvs mode Deploy a service resource and its backend pods Make real sever not ready (or may need node not ready) so as to trigger graceful delete in kube-proxy Before kube-proxy deletes the entry from ipvs, restart kube-proxy Kube-proxy becomes ready and leave the weight=0 entry as it is Service is no longer accessible through cluster ip When accessed, dmesg shows kernel messge error ofDeletion of pod will trigger kube-proxy to remove weight=0 entry from ipvs so that after the entry is recreated service will become accessible If service is 1. LoadBalancer, and, 2. .spec.externalTrafficPolicy: Local, and, 3, service is accessed from loadbalancer IP, the entry in ipvs for loadbalancer IP will only have endpoints in the same node. If backends is DaemonSet, the only one endpoints will have weight=0 thus causing connection failure After kube-proxy restarts, endpoints will be processed in the following lines. Only when the entry is in the gracefuldeleteManager.InTerminationList will be deleted. Which is not the case if kube-proxy restarts before it deletes the ipvs entry. An existing entry in the ipvs is sufficient for endpoint to be considered healthy. Weight is not considered. kube-proxy should treat ipvs entry properly by the either the following two ways. kube-proxy should initiate InTerminationList and put ipvs entry with weight=0 into the termination list kube-proxy should also look for weight in the ipvs entry instead of only UniqueRSName <details</details<detailson-premises </details<details</details<details</details<details</details<details</details
/sig network
Graceful delete of ipvs is not only triggered when pod is in terminating phase. Graceful delete seems to be triggered when endpointslice status is ready=false. This will happen when node become not ready.

However, after the node become ready again, pods on the node will only be recreated if the duration of NotReady is longer than certain time. Otherwise the pod will just become ready again. If the node will not recreate the pods && kube-proxy set the pods to the graceful delete queue && kube-proxy restarts before deletion and all cache is cleared, kube-proxy will keep ipvs entry with weight=0 even if the real server is ready. Please correct me if I misunderstand something.
/area ipvs /area kube-proxy
Seems like a legit bug to me. I can't triage until wednesday though.
I can't recreate this when I try to follow thre instructions with pod not-ready rather than node not ready. When the pod is made ready again is not stated, so I tried; a connection to a pod it not-ready. The weight is set to 0, the connection prevails -start kube-proxy. The weight is still 0, connection still working the pod ready weight is set to 1 after a while, as it should
In the instruction is different, the pod is made ready while is away. That instruction shows the fault. The reproduce instruction in this issue is incorrect.
I triage this issue as a bug, but it's really a duplicate of which was never fixed (it seems). Don't know if the proper action is to close this issue and re-open . /triage accepted
Please instruct if this issue shall be closed as duplicate and re-opened PTAL at
Thank you for your triage Monitoring IPVS weights is costly, but needed while this problem persists. Any news? Adding the following lines should fix the problem but will also add loads.
That does seem costly to do on every endpoint sync. Is there an alternative solution we can explore where we assess all endpoints once when kube-proxy initially starts up?
This is a startup-only problem. It should be solved with a one-time action.
How about modifying to be ? We could then check and update the real server weights only when is true. Only and will be modified to call , all other users will call the function with false. Since these two functions, are only called during startup, we won't be doing this monitoring every endpoint sync.
cc:


Question: } if errors.IsConflict(err) { err = fmt.Errorf(`%v Please review the fields above--they currently have other managers. Here are the ways you can resolve this warning: * If you intend to manage all of these fields, please re-run the apply

What happened: While testing out the flag, I created a manifest with the following contents: I ran , and the resources appeared on my minikube cluster: Then, I removed the deployment from the manifest: And re-ran the same command, . To my surprise, however, the deployment wasn't removed; the only output from kubectl was . What you expected to happen: I expected the pruner to find the deployment with the matching label, , and prune it. How to reproduce it (as minimally and precisely as possible): I believe my example above is minimal and easily reproducible. Anything else we need to know?: This small change to the code for resolves the issue: Environment: Kubernetes version (use ): v1.16.3 Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cli
/assign cc


Question: // node selectors labels to namespaces var NamespaceNodeSelectors = []string{"scheduler.alpha.kubernetes.io/node-selector"} <ins> var nonTerminalPhaseSelector = func() labels.Selector { var reqs []labels.Requirement for _, phase := range []v1.PodPhase{v1.PodFailed, v1.PodSucceeded} { req, _ := labels.NewRequirement("status.phase", selection.NotEquals, []string{string(phase)}) reqs = append(reqs, *req) } selector := labels.NewSelector() return selector.Add(reqs...) }() </ins> type updateDSFunc func(*appsv1.DaemonSet) // updateDaemonSetWithRetries updates daemonsets with the given applyUpdate func

master-blocking Conformance - GCE - master - kubetest2Kubernetes e2e suite. ci-kubernetes-gce-conformance-latest-kubetest2.Overall kubetest2.Test ! 2023-06-27 16:43 PDT Kubernetes e2e suite: [It] [sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance] No response /sig cloud-provider /sig apps cc
This is more critical than I expected it starts to fail in other jobs ,%20master ,%20ipvs,%20dual,%20master From the diff the best candidate seems can you PTAL /priority critical-urgent
/assign
anyone from you can please add to this issue?
/triage accepted
I have a proposed fix in . Is there a job I can add?
/remove-kind failing-test /kind flake


Question: import ( "fmt" "net" <ins> "path" </ins> "sync" "time"

/kind bug The lease reconciler creates the kubernetes endpoint with a prefix of kube-apiserver-endpoint, however that is not the prefix that the rest of kube uses. The end result is that the kubernetes service doesn't have an endpoint, and kubectl won't list the kubernetes endpoint. The lease reconciler also creates lease files with a really long lease time instead of the default 15 seconds.
/cc
cc xref
kube-apiserver-endpoint is used for the lease itself, from what I remember. I don't think that's an issue by itself.
Oh but it does look like the endpointRegistry for the lease reconciler is misconfigured to use that
i'll look into it
what is the name that should be used?
services/endpoints afaik -
Yeah, making that change locally fixed it for me
But I'm not seeing the masterleases/apiServerIPInfo data in etcd, or I'm issuing the wrong command to find it
I had that same problem. The lease timeout is set way high, but for some reason that resulted in the lease files being created for only a very short time. I'm not sure why. Playing with the ttl amount got the lease files to stay around....forever. I've got a PR I'm about to push to fix both issues.
FYI has a PR for this now
cc


Question: configs[Dnsutils] = Config{e2eRegistry, "dnsutils", "1.1"} configs[EchoServer] = Config{e2eRegistry, "echoserver", "2.2"} configs[EntrypointTester] = Config{e2eRegistry, "entrypoint-tester", "1.0"} <del> configs[Etcd] = Config{etcdRegistry, "etcd", "v3.3.10"} </del> <ins> configs[Etcd] = Config{gcRegistry, "etcd", "3.3.10"} </ins> configs[Fakegitserver] = Config{e2eRegistry, "fakegitserver", "1.0"} configs[GBFrontend] = Config{sampleRegistry, "gb-frontend", "v6"} configs[GBRedisSlave] = Config{sampleRegistry, "gb-redisslave", "v3"}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I see only amd64 architecture etcd is used for the which will create issues for the other architectures like ppc64le, arm etc. We need to replace this etcd image with the k8s maintained fat manifest image present in
/sig testing


Question: PodDeleteTimeout = 5 * time.Minute ) <del> // DeletePodOrFail deletes the pod of the specified namespace and name. </del> <ins> // DeletePodOrFail deletes the pod of the specified namespace and name. Resilient to the pod // not existing. </ins> func DeletePodOrFail(c clientset.Interface, ns, name string) { ginkgo.By(fmt.Sprintf("Deleting pod %s in namespace %s", name, ns)) err := c.CoreV1().Pods(ns).Delete(context.TODO(), name, metav1.DeleteOptions{}) <ins> if err != nil && apierrors.IsNotFound(err) { return } </ins> expectNoError(err, "failed to delete pod %s in namespace %s", name, ns) }

Which jobs are failing: Which test(s) are failing: Test Overall [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is force deleted while kubelet is down [Disruptive][Slow][LinuxOnly] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is force deleted while kubelet is down [Disruptive][Slow][LinuxOnly] [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] subPath should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow][LinuxOnly] Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /milestone v1.19 /priority critical-urgent /sig testing /sig storage
This looks like an infrastructure issue but pinging just in case anyone has immediate insight
/remove-kind bug
Looks like recent changes here: /cc
/assign
/reopen unfortunately I don't resolved the issues
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
how do you say that ? looking at grid - it looks like it is not breaking anymore? Same result with filter applied - There is one failed run after my fix merged - but that is unrelated to original PR and appears to be because of broken cluster: Did I miss anything?
apologies, I believe your fix solved its intended issue, but the job is still failing. I was not clear about that in my comment :+1:
Should this be closed - ? I do not see these tests flaking anymore.
/remove-sig testing /close
Closing this issue. <detailsIn response to : /remove-sig testing /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Dashboards are green. Wasn't SIG testing


Question: if [[ ${platform} == "linux/arm" ]]; then export CGO_ENABLED=1 export CC=arm-linux-gnueabi-gcc <ins> # See https://github.com/kubernetes/kubernetes/issues/29904 </ins> export GOROOT=${K8S_PATCHED_GOROOT} elif [[ ${platform} == "linux/arm64" ]]; then export CGO_ENABLED=1

See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question: if [[ ${platform} == "linux/arm" ]]; then export CGO_ENABLED=1 export CC=arm-linux-gnueabi-gcc <ins> # See https://github.com/kubernetes/kubernetes/issues/29904 </ins> export GOROOT=${K8S_PATCHED_GOROOT} elif [[ ${platform} == "linux/arm64" ]]; then export CGO_ENABLED=1

See
CI has built and uploaded unofficial prerelease darwin binaries for 1.4 using go1.7.1: cc Official binaries will ship as soon as we cut 1.4.1.
We have official v1.4.1 binaries available now, too.
Hooray


Question: }) } <ins> // findVersionsWithUnchangedSchemas finds each version that is in the new CRD object and differs from that of the old CRD object. // It returns a set of the names of mutated versions. // This function does not check for duplicated versions, top-level version not in versions, or coexistence of // top-level and per-version schemas, as further validations will check for these problems. func findVersionsWithUnchangedSchemas(obj, oldObject *apiextensions.CustomResourceDefinition) sets.Set[string] { versionsWithUnchangedSchemas := sets.New[string]() for _, version := range obj.Spec.Versions { newSchema, err := apiextensions.GetSchemaForVersion(obj, version.Name) if err != nil { continue } oldSchema, err := apiextensions.GetSchemaForVersion(oldObject, version.Name) if err != nil { continue } if apiequality.Semantic.DeepEqual(newSchema, oldSchema) { versionsWithUnchangedSchemas.Insert(version.Name) } } return versionsWithUnchangedSchemas } // suppressExpressionCostForUnchangedSchema returns a copy of opts with suppressPerExpressionCost set to true if // the specified version's schema is unchanged. func suppressExpressionCostForUnchangedSchema(opts validationOptions, version string) validationOptions { if opts.versionsWithUnchangedSchemas.Has(version) { opts.suppressPerExpressionCost = true } return opts } </ins> // ValidateCustomResourceDefinitionUpdate statically validates // context is passed for supporting context cancellation during cel validation when validating defaults func ValidateCustomResourceDefinitionUpdate(ctx context.Context, obj, oldObj *apiextensions.CustomResourceDefinition) field.ErrorList {

We were using cel-go v0.12.7 in Kubernetes 1.27 release and cel-go v0.16.0 in Kubernetes 1.28 release. There are many changes from cel-go side between those two releases and some may break backward compatible of CRD validation rules. An example would be the estimated cost for CRD validation rules(Thanks to who brought this to our attention with the following example). In the example CRD validation rules below: It will pass perfect in Kubernetes 1.27 version but fail in Kubernetes 1.28 with error: This is due to the estimator fix happened in cel-go(, ) The changes should not cause backward incompatible of CRD validation rules. OR a stability test suite should be in place to prevent the changes causing backward compatible issue before GA. Apply the above CRD in Kubernetes 1.27 and 1.28, 1.27 cluster should pass and 1.28 cluster should fail. Since the fix in cel-go side makes sense to me and the feature is in Beta now, I am not quite sure how strict we should be on backward compatible. cc As a followup, a stability test suite for estimate cost is worth adding before GA to make sure any future changes be caught before the feature promoted to GA. <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi I agree that the changes in cel-go should not break the backward compatibility of CRD validation rules, especially for users who have already defined and deployed their custom resources using the previous version of Kubernetes. One suggestion I have is to use a version flag for the CEL library, so that users can specify which version of cel-go they want to use for their CRD validation rules. This way, they can avoid unexpected errors or behavior changes when upgrading their Kubernetes cluster.

For example, they can add something like in their CRD schema to indicate that they want to use the old version of cel-go.
Another suggestion I have is to use a cost limit parameter for the CEL library, so that users can adjust the budget for their CRD validation rules. This way, they can fine-tune the trade-off between performance and complexity of their rules. For example, they can add something like in their CRD schema to indicate that they want to allow more expensive rules.
I agree this is the type of bug fix that is legitimate to make during beta. when the size changes came in, did our existing size estimation test coverage notice this cost increased? If not, is there a systemic change we could make to exercise each type of estimation combination so we have visibility to any future changes. Existing persisted CRD objects should continue to work as-is on upgrade, but the CEL rules and/or schema would be required to fit within fixed cost estimates before making future updates/changes to the CEL rules there should be unit tests ensuring updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates, but can you verify that is the case? That's not workable, both from an implementation perspective and from a fragmentation perspective. We explicitly don't want to allow opting into loopholes / bugs in cost estimation that allow CRD authors to make API servers do excessive amounts of work. Similar to above, the responsibility for API server resource consumption lies with the kube-apiserver admin, not the CRD author, so exposing this in the API is not something I would expect.
/assign
Thank you! I guess the next step would be: the existing test suite regarding with the cel cost evaluation. Add estimated cost tests for every single operation, function, type, etc if it's not existing now. updates to a CRD that don't change CEL rules are allowed even if the existing CEL rules exceed cost estimates. Currently it is not working like this but would definitely help here. cc , please correct me if I'm wrong :) Those should be done before the feature goes to GA. I am not sure about the policy of a GA feature relying on alpha/beta feature is acceptable or not. is going to Beta in this release. The worst case is we have to wait further until the feature is ready.

I don't think that helps... CRD ratcheting is about validation of the custom resources. This is about validation of the CRD itself.
thanks for clarifying! Are you suggesting while updating CRD, if the rules matching what's in the existing object, we should skip cost estimation check? That makes sense. I could start a PR if my understanding aligns :)
something along those lines, though now that I think about it, other changes in the schema could affect the number of times a CEL expression could be run (dropping or increasing maxItems, maxProperties, etc) we already collect preexisting expressions when updating a CRD in order to allow already-persisted expressions (possibly persisted via a newer API server) making use of new CEL features: I think that is what I was thinking of. I would probably focus on: coverage of the edges of the cost estimation so we don't pick up changes there without knowing coarse-grained skipping of schema/cel validation for a particular version in a CRD for updates which don't change anything under that version's schema (would let someone make unrelated updates to the CRD and leave an existing version's schema / CEL rules alone)
Thank you! We have those work planned for the current cycle. /assign /assign


Question: } return client.AttachToContainer(containerID, opts, sopts) } <del> func portForward(client libdocker.Interface, podSandboxID string, port int32, stream io.ReadWriteCloser) error { container, err := client.InspectContainer(podSandboxID) if err != nil { return err } if !container.State.Running { return fmt.Errorf("container not running (%s)", container.ID) } containerPid := container.State.Pid socatPath, lookupErr := exec.LookPath("socat") if lookupErr != nil { return fmt.Errorf("unable to do port forwarding: socat not found.") } args := []string{"-t", fmt.Sprintf("%d", containerPid), "-n", socatPath, "-", fmt.Sprintf("TCP4:localhost:%d", port)} nsenterPath, lookupErr := exec.LookPath("nsenter") if lookupErr != nil { return fmt.Errorf("unable to do port forwarding: nsenter not found.") } commandString := fmt.Sprintf("%s %s", nsenterPath, strings.Join(args, " ")) klog.V(4).Infof("executing port forwarding command: %s", commandString) command := exec.Command(nsenterPath, args...) command.Stdout = stream stderr := new(bytes.Buffer) command.Stderr = stderr // If we use Stdin, command.Run() won't return until the goroutine that's copying // from stream finishes. Unfortunately, if you have a client like telnet connected // via port forwarding, as long as the user's telnet client is connected to the user's // local listener that port forwarding sets up, the telnet session never exits. This // means that even if socat has finished running, command.Run() won't ever return // (because the client still has the connection and stream open). // // The work around is to use StdinPipe(), as Wait() (called by Run()) closes the pipe // when the command (socat) exits. inPipe, err := command.StdinPipe() if err != nil { return fmt.Errorf("unable to do port forwarding: error creating stdin pipe: %v", err) } go func() { io.Copy(inPipe, stream) inPipe.Close() }() if err := command.Run(); err != nil { return fmt.Errorf("%v: %s", err, stderr.String()) } return nil } </del>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Ran on windows pod and received the error message: What you expected to happen: The port be available on my localhost How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: socat is not available on windows nodes Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server version: 1.10.2 Cloud provider or hardware configuration: Azure - acs-engine OS (e.g. from /etc/os-release): Windows Server 1803 Kernel (e.g. ): Install tools: Others:
/sig windows
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I can confirm this on 1803:
What would be the approach to fixing this? If connecting to a windows machine use a different tool other than socat? Are there any tools that are used in the windows world for port-forwarding? If I got some guidance on tools to use I could take a try at a PR for this.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
The problem is that the portForward streaming operation invokes and on the node. Those aren't available (or required) on Windows.

On Linux, the command is something like
Would using be the command to get this to work? (v=ws.10)#add-v4tov4
/remove-lifecycle stale
I looked into this, I think that it could in theory work, the hard part about trying to use is that it needs the IP address, whereas in this method all you have is the PID and . I don't know how to do mapping to get the IP, but if you can figure that out then it's just a matter of allocating a random port to use as the listen port, assuming people are okay with that solution. It has the disadvantage of being less secure than the Linux side, which only exposes access to the stream to the Kubelet process instead of all processes, but I'm not sure that's a big security concern.
This function implemented by the CRI shim, which is capable of finding out the IP address by . /cc
I'm a bit ignorant of all things windows. Does "all processes" really mean everything, including unprivileged containers, or do you just mean all processes in the host network namespace (or whatever the windows equivalent is)?
It would have been just the host network namespace. This approach didn't really pan out though. is investigating another approach that more closely matches the way it works on Linux.


Question: defer c.Unlock() if !c.stopped { c.stopped = true <ins> close(c.done) </ins> close(c.input) } }

We noticed many goroutines stalled at in a large cluster (~200 nodes, ~1100 pods). It's because if the WatchServer timeout when the channel is full, will be blocked on the write to the even though is , because WatchServer stops consuming the channel after the timeout.
Great finding - thanks for debugging


Question: createVolumeConfigMap(config) By("Initializing local volume discovery base path") <del> mkdirCmd := fmt.Sprintf("mkdir %v -m 777", path.Join(hostBase, discoveryDir)) </del> <ins> mkdirCmd := fmt.Sprintf("mkdir -p %v -m 777", path.Join(hostBase, discoveryDir)) </ins> err := framework.IssueSSHCommand(mkdirCmd, framework.TestContext.Provider, config.node0) Expect(err).NotTo(HaveOccurred()) }

The local PV provisioner is being updated to discover only real mount points (and not directories), that resolves some other related issues. The related e2e tests using the local PV provisioner need to be updated accordingly.
/assign
/sig storage
FYI:


Question: return false } <del> func getServiceTags(service *v1.Service) ([]string, error) { </del> <ins> func getServiceTags(service *v1.Service) []string { if service == nil { return nil } </ins> if serviceTags, found := service.Annotations[ServiceAnnotationAllowedServiceTag]; found { <ins> result := []string{} </ins> tags := strings.Split(strings.TrimSpace(serviceTags), ",") for _, tag := range tags { <del> // Storage and Sql service tags support setting regions with suffix ".Region" if strings.HasPrefix(tag, "Storage.") || strings.HasPrefix(tag, "Sql.") { continue } if !supportedServiceTags.Has(tag) { return nil, fmt.Errorf("only %q are allowed in service tags", supportedServiceTags.List()) </del> <ins> serviceTag := strings.TrimSpace(tag) if serviceTag != "" { result = append(result, serviceTag) </ins> } } <del> return tags, nil </del> <ins> return result </ins> } <del> return nil, nil </del> <ins> return nil </ins> } func serviceOwnsPublicIP(pip *network.PublicIPAddress, clusterName, serviceName string) bool {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider has a limited support for service tags by annotations: But according to , there are actually much more service tags than this list. Hence, errors would happen if the service tag is not included in the above list. What you expected to happen: Azure cloud provider should support all service tags from Azure. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig azure /priority important-soon
/assign


Question: var _ storagelisters.CSINodeLister = CSINodeLister{} // CSINodeLister declares a storagev1.CSINode type for testing. <del> type CSINodeLister storagev1.CSINode </del> <ins> type CSINodeLister []storagev1.CSINode </ins> // Get returns a fake CSINode object. func (n CSINodeLister) Get(name string) (*storagev1.CSINode, error) { <del> csiNode := storagev1.CSINode(n) return &csiNode, nil </del> <ins> for _, cn := range n { if cn.Name == name { return &cn, nil } } return nil, fmt.Errorf("csiNode %q not found", name) </ins> } // List lists all CSINodes in the indexer.

handles cases when csiNode is nil. This is condition is used by cluster-autoscaler internal working. However, the log line refers to causing seg fault
/sig scheduling /sig storage
/sig storage /priority important-soon /triage accepted
/sig storage /sig scheduling Note that the lister returns an error for the case when the doesn't exist. But it's ignored here: It looks like the log line that causes the panic was introduced in in 1.25+
cc
/assign
As far as I can tell this issue is only fixed in version 1.27 (by ). Is there a plan to port it back to releases 1.25 and 1.26 as well?
it has already been picked to 1.25 and 1.26 () and will be in the next patch releases for those versions
Great. Thank you for the pointer.