Question: } f.pets[index].pod.Status.Phase = api.PodRunning f.pets[index].pod.Annotations[PetSetInitAnnotation] = "true" <ins> f.pets[index].pod.Status.Conditions = []api.PodCondition{ {Type: api.PodReady, Status: api.ConditionTrue}, } </ins> return nil }

This is mostly motivated by petset. Currently we create a governing service that is responsible for the network identity of each pet. The requirements of this governing service differ from a normal service in the following ways: have internal protocols to notice peer health, and a list of different checks (eg: ). In kube, peers use DNS to communicate. So we don't want a readiness check failure to remove the pet from DNS and abruptly cut it off from the set. "governing service" is headless so kube-proxy doesn't need to NAT the service ip. will not be using this Service, so it doesn't matter that unready pets show up in DNS under the svcName sub domain. In addition to these, there's one important reason some pets need to have DNS before readiness. To provide the guarantees required to safely bootstrap a pet, we need: 0 starts before pet1.. 't start pet1 till pet0 is running/ready pet1 starts, it sees pet0 in DNS The easiest way to guarantee 3 today is to do one of: poll in the entrypoint of pet0, till it sees itself poll in the init container of pet0, till it sees itself poll in the init containers of pet {1..N-1} till it sees all previous pets Third option is not a great idea because we don't know who else is supposed to exist, and for the first 2 to work with a readiness probe, we need a DNS record before the Service is ready.
I have a wip that does this through an annotation:
+1. this is useful even in traditional n rc's and n+1 services setup for clustered systems like n nodes zookeeper/etcd. Currently, there isn't a easy way to specify a readiness check in such setups, because the check in these cases would look for whether the pod member has joined the cluster successfully as follower, which wouldn't work as they discover each other through them appearing in service endpoints. this prevents one from being able to do a rolling upgrade of cluster members safely, because there is no readiness feedback after a member pod is upgraded.
Suggestion to hold this for alpha in 1.5, and in 1.6 spend a bit of time talking about how services are best intermixed with stateful sets (there's a lot of correlation between them).

That avoids us breaking people in 1.5 and gets a bit more think time.
SGTM
See also
Can we claim that if a pod is requesting pet DNS (i.e DNS name via the hostname/subdomain annotation) it never wants kube-style readiness, because it has its own internal protocol for peer health? If that's true we can get rid of the annotation and teach kubedns to always serve records for pet DNS ignoring readiness. My intuition tells me the answer to that question is no (i.e there exist pets that use DNS for discovery, and don't have an internal protocol for readiness, so they must use kube readiness to remove themselves from the peer group), and so the ability to toggle it on/off is useful, but I also want others to consider it.
I'd love to nix an annotation but I have not the expertise to answer this one.
I don't have a strong feel either way - I'd say readiness controls on a service feel orthogonal to independent names though. Since we don't have per port readiness, we need to consider that at some point we might need more sophistication on services or pods to control membership (unless we make label flipping easier). I would like to see us tease apart the "leader / follower" service member concept a bit more because that might inform how we handle readiness (seems like a separate bit, but we may want atomic changes).
They are somewhat orthogonal, but you presumably only want to discover healthy peers, unless you have a way to probe every peer you discover for health. Applying this policy based on endpoint info also feels a little weird, because it's harder to create overlapping Servics with different behavior. I think doing the common thing always and by default is fine, as long as we can annotate our way back to what we have today, but what form would the opposite of today's Service level annotation take? i.e something that says "even though my endpoints need pet names, apply the regular rediness behavior to their DNS records"
How bad would it be to start without an annotation, assume that pet-names == tolerate-unready, and see if anyone screams? Who should we consult?
SGTM, that's also what I was prposing in the previous comment
I guess the question is why do we even want to do that? Why make a change that couples logic vs leaving as is with decoupled logic?

We have people who want unready endpoints for deployments to form gossip pools - so I guess that counts as a use case where pet names is not coupled to unready. Is leaving the annotation on for another release really a problem? , Prashanth B <:
That's exactly the kind of use case I was looking for, something that doesn't require pet names. I get the feeling there will be other apps but I couldn't produce an example.
JBoss EAP uses it to form initial quorum for jgroups, which isn't a strong consensus system. So they use deployments and a minimum set to form an initial quorum and deal with split brain like jgroups normally does (infinispan has multiple consistent modes, some of which might prefer loose pools). Arguably smaller sets and some infinispan quorums might prefer statefulsets, but those use cases are also more tolerant of partitions in general. , Prashanth B <:
we are using tolerate-unready-endpoints annotation in our zookeeper deployments (not using pet/stateful set). Without unready endpoints, it is impossible to define a meaningful pod readiness check for zk, and without a readiness check, it is hard to do rolling updates (one ensemble member at a time).
The question is whether there's a need to have StatefulSets without tolerate-unready-endpoints. It sounds like maybe yes, which makes me grumpy, but I'll acquiesce if Clayton and Prashanth say so. , ravilr <:
isn't it enough to establish a need to tolerate-unready without statefulsets?
I have two services pointing to my etcd stateful set - one that is endpoints (peers) and tolerates unready. I have one that is for naive clients that uses readiness. On Dec 1, 2016, at 2:59 AM, Prashanth B <: isn't it enough to establish a need to tolerate-unready without statefulsets? — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
fine.
/sig network
Right. This is exactly why etcd operator uses the alpha feature. Can we make it beta in 1.8 release?
/cc
We had discussed promoting since we need it for StatefulSet and no one has come up with anything better.

We should make a decision and include it in our GA roadmap for workload controllers.
Be sure to include sig-network On Jul 6, 2017, at 2:40 PM, Anthony Yeh <: had discussed promoting tolerate-unready-endpoints since we need it for StatefulSet and no one has come up with anything better. We should make a decision and include it in our GA roadmap for workload controllers. — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
This is also useful for ingresses that need to retain the source IP using OnlyLocal node ports, which is important for firewalling with things like Calico. The problem that we see now is when a deployment occurs, the pod is removed from the node port before the PreStop completes, which doesn't give the upstream AWS load balancer time for the pod to fail out using normal health checks.
/close
Headless Services are a thing now


Question: "k8s.io/client-go/tools/cache" "k8s.io/client-go/tools/record" <del> "k8s.io/api/core/v1" </del> clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/kubernetes/scheme" v1core "k8s.io/client-go/kubernetes/typed/core/v1"

/kind bug /sig network What happened: Networking tight loop causing GCP Cloud rate limiting to kick in. Causing tests to fail. Suspected call stack: Looks like it could be the root cause of these test failures here: https://k8s- These storage tests are hitting rate limit on the cloud api when trying to detach/delete disks. However, we have exponential backoff for all these functions. But the rate limiting from an external source is affecting the tests. See logs snippet: Note: These logs span exactly 1 minute. Try grepping "rateLimit" or "CIDRNotAvailable" -also note that the CIDRNotAvailable error messages are probably not verbose enough as they do not pinpoint what the actual error is, you can only associate them by correlation in timing and then looking at the code. Shorter snippet from above log file to demonstrate: since this is running on hundreds of nodes hitting this API call should involve some sort of exponential backoff as well as jitter. /cc /assign
/assign
/assign
/sig cloud-provider /sig gcp /milestone v1.12 (since is marked as v1.12)
may we have a priority on this since Code Freeze is next week? :)
Quick question, are we using the default API rate limits quota for scalability projects?
I will defer to for priority because he probably has the most context on how serious this is and what it could affect
Which rate limit are you referring to? (I don't remember us changing it any time) Btw - this is the project being used for that job -
I put up a patch for adding exponential retry
Gentle reminder that for this to be considered for 1.12 past Code Freeze, which is next week, it must have a status/critical-urgent label. I see this is being actively worked on


Question: // NodeInfoSnapshot return the latest NodeInfo snapshot. The snapshot // is taken at the beginning of a scheduling cycle and remains unchanged until // a pod finishes "Reserve" point. There is no guarantee that the information <del> // remains unchanged in the binding phase of scheduling. </del> <ins> // remains unchanged in the binding phase of scheduling, so plugins in the binding // cycle(permit/pre-bind/bind/post-bind/un-reserve plugin) should not use it, // otherwise a concurrent read/write error might occur, they should use scheduler // cache instead. </ins> NodeInfoSnapshot() *internalcache.NodeInfoSnapshot // IterateOverWaitingPods acquires a read lock and iterates over the WaitingPods map.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In the new scheduler framework, plugins and scheduler core share the same , both of them use the returned by ) the in the beginning of every schedule cycle, plugins(e.g. permit plugins, bind plugins) in another goroutine might read the in the same time. Then there will be a concurrent read/write error(refer ). What you expected to happen: No concurrent read/write error happens. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig scheduling /assign /cc
That is a very good catch. The binding cycle is run in a , and so updating NodeInfoSnapshot could happen concurrently with the execution of permit/prebind/bind/postbind plugins of another pod.
There are 3 proposals to solve it: Proposal 1: Add a RWLock for the struct . It is an intuitive solution. Code need to hold a lock when visiting data of . Pros: No need to change the interface of . There should be little performance impact since most operations are read, RWLock works well for this case. But if plugins hold RLock for quite a long time, it will impact the scheduler core framework to update the , there is no promise for it. Cons: Code need to hold a lock first. And it is error prone for plugins to forget it. Proposal 2: Add other methods for interface , replacing with and , it works similar to 's two method and . Pros: Easy to use We could avoid exposing internal struct to users. Cons: Need to change the interface of . We will use RWLock to implement it. So the cons of proposal 1 also exist: If plugins hold RLock for quite a long time, it will impact the scheduler core framework to update the , there is no promise for it. Update: I checked the Golang's RWLock's implementation, it is writers-preference.

So we do not need to worry about it now. Proposal 3: Use  instead of for in the struct . Pros: is useful for this case Cons: There might be some performance impact by for some cases(Ref: , ). We need change a lot related code because 's interface is different from plain Go map. Conclusion: I prefer to choose proposal 2, because it is easier to use and it is acceptable to change interface at alpha stage, and we could find other ways to solve the .
A NodeInfo snapshot is meant to be used during a scheduling cycle, not in a bind cycle. Please refer to . It does not make sense for plugins in "binding cycle" to use the nodeinfo snapshot. A snapshot is meant to be fixed and non-changing. Binding cycle plugins should use the scheduler cache instead.
We have already that the snapshot changes after "Reserve". I think we should expand that comment and explicitly state that plugins in the binding cycle should not use the snapshot.
Documenting sounds good to me, I think the comment should explicitly warn against using it in permit/prebind/bind/postbind plugins, and mention that using it in those plugins is practically a bug.
Thanks for the explanation. It is OK if it is not intended for use by bind cycle. I'll update the document for it. BTW: I could not find any scheduler cache related code in the framework code. Anyway we can add it when it is needed.


Question: framework.ExpectEqual(len(ssList.Items), 0, "filtered list should have no Statefulsets") }) <del> ginkgo.It("should validate Statefulset Status endpoints", func() { </del> <ins> /* Release: v1.22 Testname: StatefulSet, status sub-resource Description: When a StatefulSet is created it MUST succeed. Attempt to read, update and patch its status sub-resource; all mutating sub-resource operations MUST be visible to subsequent reads. */ framework.ConformanceIt("should validate Statefulset Status endpoints", func() { </ins> ssClient := c.AppsV1().StatefulSets(ns) labelSelector := "e2e=testing"

[X] Test PR: kuberenetes/kubernetes 17 June 2021 [x] Two weeks soak end date: 1 Jul 2021 [X] Test promotion PR: kubernetes/kubernetes According to this APIsnoop query, there are still some remaining StatefulSet endpoints which are untested. Create a watch to track stateful set events Create a stateful set with a static label. Confirm that the pods are running. Get the stateful set status. Parse the response and confirm that the stateful set status conditions can be listed. Update the stateful set status. Confirm via the watch that the status has been updated. Patch the stateful set status. Confirm via the watch that the status has been patched. Using an existing as a template for a new for stateful set lifecycle test. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: # Clean out any old releases rm -rf "${RELEASE_DIR}" mkdir -p "${RELEASE_DIR}" <del> kube::release::package_build_image_tarball & </del> <ins> kube::release::package_src_tarball & </ins> kube::release::package_client_tarballs & kube::release::package_server_tarballs & kube::release::package_salt_tarball &

Due to , we no longer copy the source code into . , however. Confirmed that the file now contains nothing useful: cc


Question: if err != nil { return nil, err } <ins> // Explicitly set the GVK out.GetObjectKind().SetGroupVersionKind(gvk) </ins> return out, nil }

The admission webhook review requests seem not to include a GVK in the raw object. It's included elsewhere in the review request, so it's generally not an issue, except that the unstructured codec requires GVKs to be present (it's not a technical requirement AFAICT, just a check done in the codec logic). This means that trying to use or on an Unstructured fails. A workaround in the mean time is to use , but it would be nice if this just worked properly (fixes to the review request logic and the unstructured codec logic. <details</summary</details<details</summary</detailsserving CA and certs (at and ) webhook with CA bundle `kubectl create -f that the output object doesn't have GVK in the field Environment: Kubernetes version (use ): master-ish () /sig api-machinery
when encoded to the wire, apiVersion and kind must always be included. do you have a gist handy of what the webhook receives for a built-in type (like a pod) and a for a custom resource?
Looks like custom objects (held in unstructured objects) would include GVK, but in-tree types would not. It might be as simple as explicitly setting the GVK in ConvertToGVK: and we need to update the webhook test to ensure objects in requests always have an apiVersion/kind
fixed in
/milestone v1.14 We're entering and this looks relevant to the 1.14 release. Please if I am incorrect


Question: # TODO(piosz): remove this in 1.8 NODE_LABELS="${KUBE_NODE_LABELS:-beta.kubernetes.io/fluentd-ds-ready=true}" <ins> # To avoid running the DaemonSet on older version make sure the ip-masq-agent # only runs when the readiness label is set. NODE_LABELS="${NODE_LABELS},beta.kubernetes.io/masq-agent-ds-ready=true" </ins> # To avoid running Calico on a node that is not configured appropriately, # label each Node so that the DaemonSet can run the Pods only on ready Nodes. if [[ ${NETWORK_POLICY_PROVIDER:-} == "calico" ]]; then

https://k8s- is failing to run the follow up e2es because the ip-masq-agent daemonset is scheduling pods that get stuck in a pending state on 1.5 nodes. This short-circuits the logic for the skewed e2es, and they never run. from https://k8s-
So I see that in 1.7.0 we start setting the non-masq-cidr to 0/0 gce/config-"0.0.0.0/0" We set the addon if non-masq-cidr is 0/0 if [[ "${NONMASQUERADECIDR:-}" == "0.0.0.0/0" ]]; then setup-addon-manifests "addons" "ip-masq-agent" fi Also the agent cannot run on nodes prior to 1.7 so this seems to be correct. Once the nodes are upgraded to 1.7 then the ip-masq agent should be running and should get out of the pending state. I think this is just a general problem with the way addons are handled. I may want to enable a new addon when a master is upgraded to 1.7 by default. That addon may be a DaemonSet that wants to run on nodes meeting a certain criteria. That criteria may not be met in 1.5 but may be met when they are updated to 1.7. A few things that come to mind to fix this: -Fix the test's expectations to allow this until the nodes are updated to 1.7. -rollback the PR that made this the default: -set NONMASQUERADECIDR=10.0.0.0/8 in the test properties so that the ip-masq-agent doesn't get .
It seems like the most conservative way forward is to revert the PR (). Customers who want to use the feature can set the variable () during to be able to get the feature. What do you guys think?
I think I asked the similar question related to ip-masq-agent, but my question was dropped. At this stage, I think reverting the preferred fix.

ip-masq-agent was a daemonset introduced in 1.7 which I questioned at The problem is that after master is upgraded to 1.7, addons is trying to schedule ip-masq-agent daemonset to every node no matter which Kubelet version the node is on. The problem is that ip-masq-agent can only run on node with 1.7 release.
Even if we fix the test it doesn't fix GKE or any other environment where the master is upgraded to 1.7 and the nodes are < 1.7. I think the real problem here is that there is no distinction between staging an addon and enabling an addon. Essentially staging = running. So we end up adding labels to nodes as a way of keeping the DaemonSet disabled until those labels are to nodes. While that works fine for addons that are DaemonDets, it doesn't work for deployments and replica sets. I feel like this is going to happen again regardless of what we do here with ip-masq-agent. It also creates a secondary step. Who applies those addons? What if they forget? what if it's deleted?
Fixed by (the formatting wasn't quite right in the IR to auto-close this issue).
got reverted as it broke tests.


Question: func TestRESTgetAttrs(t *testing.T) { _, rest := NewTestREST() eventA := &api.Event{ <ins> ObjectMeta: api.ObjectMeta{Name: "f0118"}, </ins> InvolvedObject: api.ObjectReference{ Kind:            "Pod", Name:            "foo",

Using a watch expression on endpoints can lead to unpredictable behavior that includes returning a watch stream that ignores filters (e.g. a {name} determination) and returns items of the wrong type (e.g. changes to PODS in the ENDPOINTS stream). returns This result includes updates for services "kubernetes", "kubernetes-ro", "redis-slave", "redis-master", etc. Even though the requesting URL indicated only "guestbook" updates were to be watched. This seems to only occur when no resourceVersion is specified and the most recent resourceVersion has fallen off the etcd watch window. The expected behavior when resourceVersion is not specified I would think should be either: an initial update with the latest version of the resource, followed by filtered updates. nothing, followed by filtered updates. Though I'm not sure which of these is actually the spec. I would prefer (1) since it allows you to do a resource watch without having to first do a separate GET. But (2) coupled with a preceding GET would yield the same results.
IRC: cc
The pods mentioned in the stream are references to the pods corresponding to endpoints. See for discussion behind the feature. We probably need to distinguish the resourceVersion at which an object was last modified, which is also used as an update precondition, vs. the current (global) resourceVersion needed for watch.
Does the change to use GET with watch as a query parameter enable the GET to both return the latest version and subscribe to changes, or is a separate non-watch GET still required?
/subscribe
If you pass resourceVersion=0 you get list then watch (for single or multiple).
That behavior has been there for a while - the only challenge is you don't know when the list is complete, and we felt that list then watch more clearly delineated that.
This.
Watching with resourceVersion=0 should return an error, IMO.
I think the semantics for {watch a single resource} vs. {watch all resources of the same type} should be separated. When watching a single resource I would expect the watch to have consistent semantics. That is, I would expect it to be possible to watch an item and be guaranteed either: I saw every change to the item, or I knew that a change was missed (and so could do a raw GET to re-sync).

To this end it would be ideal if watch for a single item had the following semantics: Delivered the current state, followed by a list of all changes. If any change is missed or can't be delivered then the watch is terminated. Delivers either a guarantee of all changes that occurred on that resource from (exclusive) to present. If the server cannot prove that there were no missing changes then it should return an error and terminate the watch. This would imply that: Would always return an error unless the server can prove that it can deliver all changes to the resource (e.g. when the resource hasn't been created yet.) I'm surprised that the interface wants to provide multi-resource watch. It would seem that the semantics of this watch would either be potentially expensive to deliver or "best effort" such that it is basically useless for anything other than casual human consumption. Imagine a future where the master is sharded such that even different resources of the same type are on different shards. To provide meaningful semantics to a multi-resource watch would require that resource versions be meaningful in a global sense and in a way that spans all shards. Wouldn't this be undesirable? It would require that any write at any shard would need a round-trip message to all other shards to ensure that the logical clocks remain in sync. A multi-watch would similarly have to be fanned out to all shards in the backend. Both of these would defeat the benefits of sharding the data in the first place. I could understand providing a multi-resource watch for the creation and deletion of resources of a particular type (handled potentially by a secondary index in the sharded scheme) without providing a full watch of the contents. You could then use this watch to establish resource-specific watches for the resources you needed to track the contents of. Maintaining a secondary index wouldn't have to be transactional with changes that produced or destroyed items.
resourceVersion=0 is the same thing as saying watch resourceVersion=ObjectModifiedResourceVersion-1. Perhaps it should be more explicit, but it is generally useful. Why is blind watch bad? ----- Original Message -----
Also, I don't see it mentioned above but - everything in  is ignored. Confusing, but already fixed by the deprecation of .

cc
We don't offer multi-resource watch today, but in the future you're likely to correlate very strongly with a shard based on your namespace, rather than a shard on your type. I don't think sharding by type will be common except on a few types of resources (events, namespaces, possibly endpoints). The access pattern for most clients is going to be "in namespace Foo, get type X, get type Y, get type Z, get type A". Most UIs that want to watch anything will watch on the namespace. So the access pattern is less likely to reward sharding by type.
The query: Is a multi-resource watch. It watches for all resources of type Endpoints. But do you want the interface to guarantee that all objects in the same namespace are updated with consistency? I agree that shard boundaries are almost certainly NOT type-boundaries. For instance there is clearly a consistency relationship between a Service and its Endpoints object. I would hope that it would be impossible for me to observe a change to a Service object and then to subsequently query for its Endpoints object and get a stale read (semantically speaking). But that is a query across types. Suggesting that the appropriate shard boundaries are certainly not type-boundaries.
Sorry, I was referring to "multi-resource" as "watching over multiple resource types". It is possible to emulate a watch across a small number of stable shards by creating a resource version tuple (a vector clock, essentially) - for medium sized clusters I see that as a reasonable option. It's more likely though that infrastructure components that want to "watch the world" are either themselves shardable (such that it's acceptable for world to be defined as "my shard") or an intermediate component supports them. That intermediate component might be something like a partially ordered event stream that provides a firehose to the cluster as an aggregator of the cluster changes. Today, we require that API consumers assume resource version is internally consistent across a single type in a single namespace. Controllers are allowed to watch the world until such a time as we implement a sharding solution. There is no consistency relationship across resource types - services and endpoints are independent objects with independent lifecycles. It is by design that Endpoints can be updated out of band with Services, to allow downstream components to offload them.

We relax referential rules across resource types - there is no guarantee a referenced object will be created before or after you, or that it should exist only if you exist.
I completely agree there are possible implementations that can tolerate sharded and/or inconsistent queries. And deciding that certain kinds of JOINs, particularly across resource types and/or namespaces, are always going to be inconsistent seems like a fine place to land. I was merely surprised that the API was designed to promote the appearance of consistency for these queries today. I would have expected that the API would be more conservative to reserve these degrees of freedom around consistency for future implementations without requiring a future breaking change. Offering less functionality in V1 or functionality with weaker semantics would allow weakly consistent future implementations to emerge later without breaking existing applications. As a general rule I try to avoid APIs that give the appearance of consistency if they are not intended to actually deliver consistency. I think most developers will always assume consistency unless there is a really good reason to believe otherwise. This seems like human nature since people expect systems to exhibit the same causal behavior as the real world. Take for instance Endpoints. My impression of this after seeing the API was that Endpoints was simply a VIEW (in the SQL since) that provides a convenient way to express the JOIN between Service and Pods. From that perspective I would certainly have expected there to be a causal relationship between a change in the Service and its corresponding Endpoints object. I could easily imagine myself making that mistake when writing a application. Perhaps my program changes the Service object and then wants to send a message to all of the Endpoints associated with the new Service definition. After changing the Service, my program queries Endpoints. Did it get the new set of Endpoints? There are several possible outcomes: The Endpoints is the same as it was before I changed the Service Definition: The new Service definition just happens to result in the same set of Endpoints as before. I have not waited long enough for Endpoints to be updated. How long should I wait? What indicates that I've waited long enough? The Endpoints is different: Is this the correct one? The one that corresponds to my Service definition change? Or is this the update that occurred due to a different Service definition change that happened just before mine? If so, I should wait longer for mine.

Or is this the update that occurred due to a different Service definition change that happened just after mine? If so, I should either use this one or attempt my Service definition change again. In an inconsistent and asynchronously updated world we need to provide a way to answer these questions. (e.g. perhaps Endpoints should include the the resourceVersion of the Service definition used to compute it.) All applications need to properly deal with this extra information. (e.g. need to ignore all updated Endpoints until one arrives that has a resourceVersion = their change to the Service definition.) I'd imagine it's possible to implement this and for customers to use it correctly, but I think it offers a lot of opportunity for bugs. Users might simply wait for the next update and then take the wrong action leading to a misbehaving system. If we could decide which relationships should exhibit consistency and provide it where the developer would expect it then the API is simply and easier to use and applications are more likely to be correct. If we don't offer APIs which might lead the developer to make incorrect assumptions then applications are less likely to have those bugs.
I dug into this... it's more broken than I thought it was, so I've made a fix in . Once my fix goes in, a single object acts as if you didn't pass a name: this will be fixed directly (at least for the resource types I checked). get an appropriate resource version, you will have to LIST with a fieldselector ("?fields=name%3Dfoo" or some such-- don't recall exact syntax) instead of GETting. This way we don't have to modify the API to pass a special "watch at" resource version.


Question: "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" <ins> utilfeature "k8s.io/apiserver/pkg/util/feature" </ins> coreinformers "k8s.io/client-go/informers/core/v1" clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/kubernetes/scheme"

For various reasons (network topology, heterogenous nodes, etc) a user may want to control the addition of a node to a cloud load balancer. A specific issue with the Azure Container Instances connector which introduces a virtual node can be found here: The right way to do this is via a node annotation, e.g.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
/reopen This is still a problem.. any word on when feature gate goes out of alpha?
Setup the enhancement tracking issue for this Still discussing with sig-network on how we want to move forward with this.


Question: t.Errorf("expected success or bad request err, got %v", err) } }) <ins> t.Run("MergePatchType should handle a valid patch just under the max limit", func(t *testing.T) { patchBody := []byte(`{"value":0` + strings.Repeat(" ", 3*1024*1024-100) + `}`) err = rest.Patch(types.MergePatchType).AbsPath(fmt.Sprintf("/api/v1/namespaces/default/secrets/test")). Body(patchBody).Do().Error() if err != nil { t.Errorf("unexpected error: %v", err) } }) </ins> t.Run("StrategicMergePatchType should handle a patch just under the max limit", func(t *testing.T) { patchBody := []byte(`{"value":` + strings.Repeat("[", 3*1024*1024/2-100) + strings.Repeat("]", 3*1024*1024/2-100) + `}`) err = rest.Patch(types.StrategicMergePatchType).AbsPath(fmt.Sprintf("/api/v1/namespaces/default/secrets/test")).

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: It returns an error when I create a pod of yaml file over 1M. I checked the codes and found that the coded wrong. It should be “json.Unmarshal(p.patchBytes, &v)”, not “json.Unmarshal(p.patchBytes, v)” in the 344 and 368 line. What you expected to happen: It can be parsed correctly. How to reproduce it (as minimally and precisely as possible): Look at the code and you'll see or you can test it. Anything else we need to know?: no. Environment: Kubernetes version (use ): Community latest code Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/kind bug
/sig api-machinery
/assign


Question: package registry import ( <ins> "fmt" </ins> "net" "strconv" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util" </ins> "github.com/golang/glog" )

What would you like to be : We'd like to have the Prometheus stack () running in our scalability tests. Why is this needed: There are couple of reasons: will simplify writing SLI assertions/tests that are based on metrics not necessarily exported on master. Some examples here are the new In-Cluster Network Programming Latency SLI (exported in kube-proxy) or soon to be implemented RTT SLI where the metric will be exported in regular pods (probes). With Prometheus stack, collecting and aggregating such metrics is very simple. -prometheus provides many useful metrics about the state of the cluster, such metrics might be invaluable when debugging scalability test failures. mimics what customers do in their cluster, making the test setup more realistic.
/sig scalability
/assign
The GCE 5K Node Performance test with enabled Prometheus completed successfully. We can enable Prometheus in all tests.
Looks like kubemark-500 has some issue, test fails often with Kubemark-5000 also failed once with Will disable Prometheus in kubemark, will work on fixing it when I'm back from my vacation.
For the first kind of the failure that kubemark-500 suffered from, i.e. the root cause is Looks like the token for the "prometheus-operator" service account was not ready, as we can see in the logs it was created after the POST to create the servicemonitor was called.
It looks like the second failure (kubemark-5000) was caused by the prometheus-k8s pods not scheduled by the kube-scheduler. This is strange but may be related to other issues we had. Hard to investigate what really happened there, will leave it for now. Will re-enable prometheus tomorrow and will see if it happens again.
Closing, as this is basically done in a minimal version. Other improvements will be tracked in separate issues. /close
Closing this issue. <detailsIn response to : Closing, as this is basically done in a minimal version. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: fi # Network plugin if [[ -n "${NETWORK_PROVIDER:-}" ]]; then <del> if [[ "${NETWORK_PROVIDER:-}" == "cni" ]]; then flags+=" --cni-bin-dir=/opt/kubernetes/bin" else flags+=" --network-plugin-dir=/opt/kubernetes/bin" fi </del> <ins> flags+=" --cni-bin-dir=/opt/kubernetes/bin" </ins> flags+=" --network-plugin=${NETWORK_PROVIDER}" fi if [[ -n "${NON_MASQUERADE_CIDR:-}" ]]; then

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare-metal OS (e.g. from /etc/os-release): Ubuntu 14.04 Kernel (e.g. ): Linux dev-k8s-controller-p1 4.4.0-53-generic ~14.04.1-Ubuntu SMP Fri Dec 2 03:43:31 UTC 2016 x8664 x8664 x86_64 GNU/Linux Install tools: chef cookbook Others: Docker 1.12.6, calico 2.1 CNI binaries: kubelet config: What happened: When trying to run pod error message is generated: So it seems that kubelet is trying to find CNI binaries in paths: /opt/loopback/bin /etc/cni/net.d /opt/calico/bin But it should look at since this is the default valur for parameter. What you expected to happen: Pod starts. How to reproduce it (as minimally and precisely as possible): Run pod: with kubelet 1.6 and CNI binaries in those path (default for calico): Anything else we need to know: After copy to and pod starts fine.
Good, very helpful to me
When set and not set , default path of CNI binaries will not take effect, the path specified by will be used instead. We should use and instead of if use CNI plugin.
I will push a PR to deprecate the flag. It is confusing. Closing this issue.
Was that ? It was closed without merging. I agree should be deprecated.
I think is needed before is removed. The flag is still not deprecated, I will send a PR to deprecate it.


Question: const isInvalidQuotaResource string = `must be a standard resource for quota` const fieldImmutableErrorMsg string = genericvalidation.FieldImmutableErrorMsg const isNotIntegerErrorMsg string = `must be an integer` <del> const isZeroErrorMsg string = `must be greater than zero` </del> <ins> const isNotPositiveErrorMsg string = `must be greater than zero` </ins> var pdPartitionErrorMsg string = validation.InclusiveRangeError(1, 255) var volumeModeErrorMsg string = "must be a number between 0 and 0777 (octal), both inclusive"

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT /kind bug What happened: If someone requests a zero (0) capacity PVC and an administrator creates a zero (0) capacity PV, Kubernetes will happily bind them together. Unfortunately, any use of this PVC/PV combination by a pod will fail. It fails because the zero capacity PV cannot be formatted or mounted successfully. Note: The same issue happens for dynamically provisioned PVs. What you expected to happen: I expect that Kubernetes would not allow creation of invalid PVC and PV objects, aka zero (0) capacity PVs and PVCs. How to reproduce it (as minimally and precisely as possible): Create a pod that requests a zero capacity PVC, using dynamic PV provisioning is simpler than creating your own zero (0) capacity PV. The pod will fail to reach a running state, since the related PV fails to be formatted and mounted successfully. Here is the error that I received for a test I performed requesting a zero capacity PVC using ceph rbd: Anything else we need to know?: Here is a related item with a specific fix for external ceph rbd provisioner: Environment: Nothing specific to environment
/sig storage
/assign


Question: "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" utilfeature "k8s.io/apiserver/pkg/util/feature" <ins> "k8s.io/client-go/dynamic" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" restclient "k8s.io/client-go/rest"

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

[x] Original approval issue: Start 9 June 2020 3 Flakes No Flakes for 2 weeks According to this APIsnoop query, there are still some remaining ReplicationController endpoints which are untested. a ReplicationController with a static label the ReplicationController with a new Label and updated data the ReplicationController's status /Scale the ReplicationController's Replica count to 2 Replicas the ReplicationController the ReplicationController's Status all ReplicationControllers in all Namespaces find the ReplicationControllers(1) ensure that the ReplicationController is found and is patched Namespaced ReplicationControllers(1) via a Collection with a LabelSelector Discover useragents: See all endpoints hit from this test Display test coverage change If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
Can you please add in PR
/reopen Merged. Promotion to be created
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" utilfeature "k8s.io/apiserver/pkg/util/feature" <ins> "k8s.io/client-go/dynamic" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" restclient "k8s.io/client-go/rest"

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

[x] Original approval issue: [x] Two weeks soak end date: 8 May 2020 [x] Two weeks soak end date: 23 June 2020 This issue is created to allow edit by /wip /hold This issue proposes the writing of a new test for the API endpoints , with the intention of promoting it to Conformance. This issue contains a very basic mock test in order to start the conversation about what a good e2e test would look like for this endpoint. The code below is verified to be hitting the intended endpoint, show as per the queries in APIsnoop's live view of the cluster's audit logs. According to this APIsnoop query, there are still some remaining ConfigMap endpoints which are untested. a ConfigMap with a static label the ConfigMap with a new Label and updated data the ConfigMap to ensure it's patched all ConfigMaps in all Namespaces find the ConfigMap(1) ensure that the ConfigMap is found and is patched Namespaced ConfigMap(1) via a Collection with a LabelSelector This mock test uses client-go's standard functions for talking to the Event area of the API. Discover useragents: Display endpoints hit by the test: Show the change in coverage after the mock test: If a test with these calls gets merged, Conformance coverage will go up by 2 points
/sig testing /sig architecture /area conformance
/assign Issue created to replace
GitHub didn't allow me to assign the following users: raymonddeng99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen On hold & soaking
Reopened this issue. <detailsIn response to : /reopen On hold & soaking Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" utilfeature "k8s.io/apiserver/pkg/util/feature" <ins> "k8s.io/client-go/dynamic" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" restclient "k8s.io/client-go/rest"

This Issue replace Watch events may be missed and the test may still pass. We need to ensure that all expected watch events are seen. Watch events are not collected and checked at the end of the test. In order to solve the problem, we must collect all watch events as they come in and perform a check at the end of the test to ensure that the order is correct and they contain no errors.
/sig testing /area conformance /assign /cc
This Issue replace
AI: Must allow for other events to occur through the expected watch events stream expected watch events: must allow the following to be valid:
[x] Set up a watch for resource through a static label to the resource with DynamicClient [x] Manage the watch if it dies [x] Feed the watch to the function [x] Read all the watch events from the at the end of the test, ensuring that the events occur question: What does doing this mean to tests which check data? (e.g: - waits for values, sourced from watch) [x] Retry the test, upon failure [x] Manage the resources on retry (delete)
More or less, yes. If we need to read results mid-test, I think that will be OK, rather than waiting for the end. The main point is to wrap the watch such that it gets restarted if it fails. That wrapper will need to sit between the raw watch channel and the test: raw watch -channel -wrapper (restarts watch if closes/fails prematurely) -channel consumed in test I'm a little surprised there isn't already a sort of "buffered watch" object that does this already ? You'll also need a stop channel for the go routine started by the wrapper, so that at the end of the test you can stop that go routine and watch.
Also, if we have the "protected watch" wrapper, not sure we need to retry the test on failure. AFAIU, the only reason we wouldn't see the watch events we expected would be because of etcd compaction, which would not be expected to happen in e2e clusters, and even if it did it would not happen to event histories that are brand new (i.e., it only compacts old history). Again would like to confirm.
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen While we did push through this first iteration, this ticket is for ongoing discussion of watch tooling and refactoring of the approach.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // cleanupPodsInNamespace deletes the pods in the given namespace and waits for them to // be actually deleted. func cleanupPodsInNamespace(cs clientset.Interface, t *testing.T, ns string) { <del> if err := cs.CoreV1().Pods(ns).DeleteCollection(context.TODO(), metav1.DeleteOptions{}, metav1.ListOptions{}); err != nil { </del> <ins> if err := cs.CoreV1().Pods(ns).DeleteCollection(context.TODO(), *metav1.NewDeleteOptions(0), metav1.ListOptions{}); err != nil { </ins> t.Errorf("error while listing pod in namespace %v: %v", ns, err) return }

What happened: Scheduler decided that it wants to schedule Pod P on Node N after preempting some pods from N. It set the P.nominatedNodeName to N. After a while (before scheduling actually happened) node N was removed from the cluster. Scheduler did not reset P.nominatedNodeName to empty value. What you expected to happen: I would expect scheduler to unset P.nominatedNodeName when the node is no longer valid. The trivial reason for node not being valid any more is it is removed from the cluster. But there are more cases to cover. Out of the top of my head I may think of: extra Pod was scheduled to node and there is not enough resources any more either scheduled by scheduler or directly as static pod/daemonset node became unschedulable How to reproduce it (as minimally and precisely as possible): We did not try to reproduce issue manually. Yet following setup should work. create cluster with nodes A and B. ensure all system pods are running on A. Fill up A so it does not have spare resources. add a low priority Pod L with long termination, with requests matching allocatable of B. It should be scheduled to B. add a high priority Pod H. It should get B as nominatedNodeName kill node B the H.nominatedNodeName will still point to B which is no longer in the cluster Anything else we need to know?: Current behavior does not play well with Cluster Autoscaler which assumes that Pod with nominatedNodeName will be scheduled in a while. Therefore such pods do not trigger scale-up. If we can have P.nominatedNodeName set incorrectly for unbounded period of time CA will never provision node for P to run. Environment: Kubernetes version (use ): We observed the behavior during presubmit tests which run k8s built from HEAD of master on 27.11.2019.

Versions logged in test log: I1125 18:55:11.691] Client Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1183+", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-25T17:04:18Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} I1125 18:55:11.692] Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.0-alpha.0.1183+", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-11-25T17:04:18Z", GoVersion:"go1.13.4", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GCE /sig scheduling
/assign
/cc
Another interesting case would be creating a new pod with pod antiaffinity blocking the pod with nominated node name from scheduling: up a cluster with pods, including a long-terminating low-priority pod A. high-priority pod B that has the same requests as A. pod C with negligible resource requests and zonal topology pod antiaffinity matching pod B (and no other pods in the cluster). What I'd expect: Pod C keeps pending, nominatedNodeName is cleared. Pod A is restarted on the same node it was running before. I haven't actually tested this scenario, so I'm not sure what will actually happen. If report is correct, this may be a more convenient way of reproducing the issue (no need to add / remove nodes, it's not important which pod will go to which node).
I'm not sure if we need to clear nominatedNodeName on Pod when node is no longer valid immediately. And we have other logic to clear nominatedNodeName. Reuse your example. create cluster with nodes A and B. ensure all system pods are running on A. Fill up A so it does not have spare resources.

add a low priority Pod L with long termination, with requests matching allocatable of B. It should be scheduled to B. add a high priority Pod H. It should get B as nominatedNodeName kill node B the H.nominatedNodeName will still point to B which is no longer in the cluster After this, The H will be scheduled again. Obviously it's going to fail. H will enter the preemption process and the is empty. It will return itself as preemption process will clean nominatedNodeName on the Pod H.
when I have some time I will check if reproduction steps I describe actually work. But we surely observed pods with lingering nominatedNodeName which pointed to inexistent node, during our E2E tests. Quick thought. The method has many return points. If it returned at the place you linked it seems should be cleared. But there are many return points below, which set the list of pods to be cleared to nil. Is it possible that for a pod to consistently exit from with one of those return points?
I think you are right. Actually there are many return points to return nil. I think maybe we need to clear the of the pod after the filter failed again and before the new preemption process to avoid potential risks. WDYT?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/remove-lifecycle stale
I believe it's a bug as in old logic, e.g., v1.18.3, if the Pod (with nominatedNodeName set) cannot be scheduled and also preemption doesn't help ( nil potentialNodes is a corner case, it's more common the pod can't fit on all potentialNodes), it won't hit L413: then in old recordingFailure logic, nominatedNodeName remains as is. In the latest codebase, introduced a fix to combine setting nominatedNodeName as well update the pod's unschedulable condition. However, this doesn't help in this issue either as it ignores the case when the pod's nominatedNodeName is nil:
/reopen as regresses - we need to come up with a complete fix.
Reopened this issue. <detailsIn response to : /reopen as regresses - we need to come up with a complete fix. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle rotten
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
what are the implications of not clearing the nominated node name?
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ``` --flatten[=false]: flatten the resulting kubeconfig file into self contained output (useful for creating portable kubeconfig files) <del> --merge=true: merge together the full hierarchy of kubeconfig files </del> <ins> --merge[=true]: merge together the full hierarchy of kubeconfig files </ins> --minify[=false]: remove all information not used by current-context from the output --no-headers[=false]: When using the default output, don't print headers. -o, --output="": Output format. One of: json|yaml|wide|name|go-template=...|go-template-file=...|jsonpath=...|jsonpath-file=... See golang template [http://golang.org/pkg/text/template/#pkg-overview] and jsonpath template [http://releases.k8s.io/HEAD/docs/user-guide/jsonpath.md].

It acts like a boolean flag but requires an argument, unlike boolean flag is pretty much every environment. We need to find a better way to express the things that this is expressing. Maybe a add a tristate to the pflag lib that behaves the same way but doesn't demand an argument?
cc
The handles "true", "false", "unset" and has a default value. I seem to recall you were looking at changing it, but we needed to preserve the value. The complain here is that you have to do instead of just . /cc
fixing is really easy....


Question: "properties": { "maxUnavailable": { "type": "string", <del> "description": "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding up. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods." </del> <ins> "description": "The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. This can not be 0 if MaxSurge is 0. By default, a fixed value of 1 is used. Example: when this is set to 30%, the old RC can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old RC can be scaled down further, followed by scaling up the new RC, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods." </ins> }, "maxSurge": { "type": "string",

API documentation says is rounded up, but it is actually rounded down: The code shows it is rounded down, not up: and
Yes, the documentation is obsolete. We changed maxUnavailable to round down so that we wouldn't accidentally infer 100% unavailability for non-100 percentages. For example, maxSurge=25% and maxUanavailable=25% with a Deployment of 1 replica meant that we could scale down 1 before thus leading in 100% maxUnavailable (implicit).


Question: } if pvc.Spec.DataSource != nil { w.Write(LEVEL_0, "DataSource:n") <del> w.Write(LEVEL_1, "Name:t%vn", pvc.Spec.DataSource.Name) </del> <ins> if pvc.Spec.DataSource.APIGroup != nil { w.Write(LEVEL_1, "APIGroup:t%vn", *pvc.Spec.DataSource.APIGroup) } </ins> w.Write(LEVEL_1, "Kind:t%vn", pvc.Spec.DataSource.Kind) <ins> w.Write(LEVEL_1, "Name:t%vn", pvc.Spec.DataSource.Name) </ins> } printPodsMultiline(w, "Mounted By", mountPods)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Miss APIGroup of DataSource. What you expected to happen: kubectl describe pvc should also describe the apigroup How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig storage


Question: var timeCh <-chan time.Time doneCh := ctx.Done() <ins> if !sliding { timeCh = t.C() } </ins> // if immediate is true the condition is // guaranteed to be executed at least once, // if we haven't requested immediate execution, delay once

While trying to update to use latest version of package and to migrate away from deprecated , function, our tests fail as it seems new function immediately executes given condition twice, which is not mentioned anywhere. Result of executed test specified below: Since has been deprecated and it recommends using , I expected it to have the equivalent semantics to allow easy migration from deprecated functions, as no changelog or documentation mentions new semantics. Playground: Local: file with the following content: as . No response_ <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
CC since you were involved in
/assign /triage accepted
I'd like to work on this one if it's allowed I think this is a simple fix with few lines removed: i.e.: instead of only wait a loop if immediate is false, we wait a time loop after special execution also if the immediate is true.
/assign
/assign


Question: }) ginkgo.Context("and the container runtime is temporarily down during pod termination [NodeConformance] [Serial] [Disruptive]", func() { <ins> ginkgo.BeforeEach(func(ctx context.Context) { // Ensure that prior to the test starting, no other pods are running or in the process of being terminated other than the mirror pod. // This is necessary as the test verifies metrics that assume that there is only one pod (the static pod) being run, and all other pods have been terminated. gomega.Eventually(ctx, func(ctx context.Context) error { podList, err := e2epod.NewPodClient(f).List(ctx, metav1.ListOptions{}) if err != nil { return fmt.Errorf("failed listing pods while waiting for all pods to be terminated: %v", err) } var remainingPods []string for _, pod := range podList.Items { // The mirror pod is the only expected pod to be running if pod.Name == mirrorPodName && pod.Namespace == ns { continue } remainingPods = append(remainingPods, fmt.Sprintf("(%s/%s)", pod.Namespace, pod.Name)) } if len(remainingPods) > 0 { return fmt.Errorf("not all pods are terminated yet prior to starting mirror pod test: %v pods that still exist: %v", len(remainingPods), strings.Join(remainingPods, ",")) } return nil }, f.Timeouts.PodDelete, f.Timeouts.Poll).Should(gomega.Succeed()) }) </ins> ginkgo.It("the mirror pod should terminate successfully", func(ctx context.Context) { ginkgo.By("verifying the pod is described as syncing in metrics") gomega.Eventually(ctx, getKubeletMetrics, 5*time.Second, time.Second).Should(gstruct.MatchKeys(gstruct.IgnoreExtras, gstruct.Keys{

Example failure - Serial node e2e tests This is new test introduced in v1.27 No response No response No response /sig node
It seems the issue is that a previous test that ran had a pod around which was not terminated, namely The test started around but (which was created by pod-conditions-test), was fully terminated only at which is after this test started: Due to the pod not being fully terminated when the test is being run, the metric assertions fail.
/triage accepted


Question: <del> /* Copyright 2016 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cmd import ( "bytes" "fmt" "io" "os" "github.com/spf13/cobra" "k8s.io/apimachinery/pkg/apis/meta/v1" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/kubectl/cmd/util/editor" "k8s.io/kubernetes/pkg/kubectl/resource" ) // NewCmdEditConfigMap is a macro command to edit config maps func NewCmdEditConfigMap(f cmdutil.Factory, cmdOut, errOut io.Writer) *cobra.Command { options := &resource.FilenameOptions{} cmd := &cobra.Command{ Use:     "configmap", Aliases: []string{"cm"}, Short:   "Edit a config map object.", Long:    "Edit and update a config map object", Run: func(cmd *cobra.Command, args []string) { RunEditConfigMap(cmd, f, args, cmdOut, errOut, options) }, } addEditFlags(cmd, options) cmd.Flags().String("config-map-data", "", "If non-empty, specify the name of a data slot in a config map to edit.") return cmd } // RunEditConfigMap runs the edit command for config maps. It either edits the complete map // or it edits individual files inside the config map. func RunEditConfigMap(cmd *cobra.Command, f cmdutil.Factory, args []string, cmdOut, errOut io.Writer, options *resource.FilenameOptions) error { dataFile := cmdutil.GetFlagString(cmd, "config-map-data") if len(dataFile) == 0 { // We need to add the resource type back on to the front args = append([]string{"configmap"}, args...) return RunEdit(f, cmdOut, errOut, cmd, args, options) } cmdNamespace, _, err := f.DefaultNamespace() if err != nil { return err } cs, err := f.ClientSet() if err != nil { return err } configMap, err := cs.Core().ConfigMaps(cmdNamespace).Get(args[0], v1.GetOptions{}) if err != nil { return err } value, found := configMap.Data[dataFile] if !found { keys := []string{} for key := range configMap.Data { keys = append(keys, key) } return fmt.Errorf("No such data file (%s), filenames are: %vn", dataFile, keys) } edit := editor.NewDefaultEditor(os.Environ()) data, file, err := edit.LaunchTempFile(fmt.Sprintf("%s-edit-", dataFile), "", bytes.NewBuffer([]byte(value))) defer func() { os.Remove(file) }() if err != nil { return err } configMap.Data[dataFile] = string(data) if _, err := cs.Core().ConfigMaps(cmdNamespace).Update(configMap); err != nil { return err } return nil } </del>

kubectl edit is not working on HEAD. It's working on 1.5.


Question: switch testcase.op { case "Supports": if supports, _ = fakeSupportsQuotas(testcase.path); supports { <del> t.Errorf("Case %v (%s, %v) supports quotas but shouldn't", seq, testcase.path, false) </del> <ins> t.Errorf("Case %v (%s, %s, %v) supports quotas but shouldn't", seq, testcase.name, testcase.path, false) </ins> return true } return false

When sizeLimit is specified subsequent invocations will fail with the message: No error should be raised. Create a pod with emptyDir and sizeLimit with FSQuotaMonitoring enabled, MountVolume.SetUp is called again only if the first iteration takes longer then the actual reconciler loop interval (100ms). No response <detailsversion.Info{Major:"1", Minor:"27+", GitVersion:"v1.27.0-beta.0.87+", GitCommit:"", GitTreeState:"clean", BuildDate:"2023-04-10T09:05:56Z", GoVersion:"go1.20.2", Compiler:"gc", Platform:"linux/amd64"} </details<detailsSelf-managed cluster in Azure </details<detailsNAME="Red Hat Enterprise Linux" VERSION="9.1 (Plow)" ID="rhel" IDLIKE="fedora" VERSIONID="9.1" Linux test-master-0 6.3.0-rc3 SMP PREEMPTDYNAMIC Thu Apr 6 20:44:15 UTC 2023 x8664 x8664 x8664 GNU/Linux </details<detailsContainerd 1.6.15 </details<detailsCNI: CANAL 3.24.1 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
May you please share more information? Which version of Kubernetes you are using? OS version etc.
The problem occurs with the latest master branch, I updated the initial post with versions.
Thank you.


Question: test.cachedItem.fit, test.cachedItem.reasons, test.equivalenceHashForUpdatePredicate, <ins> true, </ins> ) // if we want to do invalid, invalid the cached item if test.expectedInvalidPredicateKey {

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)


Question: }) if !kl.standaloneMode { <del> hostIP, err := kl.GetHostIP() </del> <ins> hostIP, err := kl.getHostIPAnyWay() </ins> if err != nil { glog.V(4).Infof("Cannot get host IP: %v", err) } else {

Currently we query the apiserver for the node (indirectly: ) before populating hostip on a pod. I believe this is because the node controller is responsible for calling into the cloud provider and populating the ip address on a node: This is disappointing for 2 reasons: makes using ips to identify nodes harder (). 't separate concerns sufficiently. If the reflector fails to watch nodes reliably pods won't get a hostip.
Also see .
I would argue that if we're unable to watch nodes reliably, we have many other problems. That seems like something we need to fix, rather than work around.
watch is watch, a separation of concerns between the kubelet (which is running on the node that has the hostip) and the apiserver is a different problem imo.
After discussion, we decided to just remove host IP from PodStatus. (It will still be available from "kubectl get nodes" if someone needs it.) does that sound OK with you?
I'm working on making the nodes register themselves rather than having the node controller do it for them (see which should turn into a PR later this week). Will that assuage your concerns here?
I hadn't looked at your PR. We discussed having Kubelet pull ths information from the cloud provider but thought it might not be a good idea due to (1) requiring Kubelet to understand the details of all the cloud providers, and (2) making very hard to rate limit the calls into the cloud provider. I could imagine an approach where we still use NodeController to get the node address, but the node still initiates the registration into the cluster. Not sure whether that's a good idea, though.
Kubelet won't necessarily (and really shouldn't) have the credentials necessary to contact the cloud provider. I don't remember why HostIP was to PodStatus, and I would favor removing it, except that it would be a breaking change. You could ask on google-containers@ whether anyone is relying upon it. HostIP is tricky, as hosts may have multiple addresses, in general, which is why NodeAddresses is now an array.
On GCE / AWS do you even need credentials to determine your external IP? I thought you got it from the metadata server running locally without needing to hit an Auth'd API endpoint.
HostIP wasn't the external IP AFAIK.

It's not well defined what it is, which is one reason it should be eliminated. Yes, one should be able to get the host IP from metadata, but the cloudprovider library doesn't currently do that.
I'm also in favor of removing HostIP from pods. But I don't agree with the reason in the OP (that sometimes it'll be blank when kubelet hasn't heard from apiserver)-- I think that's par for the course.
Should we schedule a meeting to discuss whether it's reasonable for Kubelets to pull their IPs directly from the cloud provider? What is the alternative -- a new API endpoint on the master where unregistered nodes can query their IP address from NodeController? Do nodes really need their IP address when they register with the master?
That sounds like a good idea (I think we can likely clear this up with a quick meeting rather than going back and forth for days in github comments).
We all agreed that we should remove HostIP from pods. Nodes do not need to provide their address(es) when they register. It's not part of the spec. They could provide whatever hostnames and/or addresses they can get through normal Linux mechanisms and/or could contact a local metadata server, but I don't think that's necessary, since the node controller will update the addresses once the node is registered, unless we want to support scenarios with no cloud provider and no external node database. In any case, we shouldn't discuss self-registration in this issue. There are other issues that cover that topic.
OK, so this issue just boils down to removing HostIP from PodStatus, and we'll discuss the rest in ?
Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to and .
Today Kubelet fills PodStatus.HostIP in generatePodStatus() in Master (e.g. NodeController) is not involved. Currently the test waits two minutes for a pod to get a HostIP Considering that we have proposed to remove this field completely, I think we should just bump up the timeout a bit and see if that fixes the test. Any other thoughts? I have also sent out email to the appropriate mailing lists to see if anyone is using PodStatus.HostIP, since we would like to get rid of it, as discussed upthread.

cc/
BTW the title of this issue says "Kubelet shouldn't require apiserver to populate host ip of pods." But it wasn't clear to me where the dependency on the API server is (unfortunately 's original comment points to a very old revision to so I'm not sure which line he was referring to). The work of finding the HostIP seems to be done by Kubelet.GetHostIP() / Kubelet.GetNode() which assumes Kubelet.nodeInfo is filled in -- is that the dependency? I couldn't find where that gets filled in. would be great if you could shed some light.
I haven't looked at this code recently but nodinfo appears to still be retrieved from a cache populated via apiserver watch:
Ah yes, that looks right, thanks (well, "right" in the sense of "that's where it gets populated," not in the sense of "it makes sense that kubelet is using code defined in the scheduler predicates for this"...) while you're here, do you have a suggestion of the best course of action? Long-term we want to get rid of PodStatus.HostIP, but short-term we need to do something about this test. How can we make populating the cache more reliable? Or should we just increase the timeout on the test? Or something else?
It looks like the kubelet already has the node address: , so assuming no one else is going to change the ip, we could just store it in the kubelet and add it for every pod? Unsure if this is a safe assumption.
I also haven't debugged this new set of test failures, so I'm not sure if it's failing for the same reason
... to make on all clouds
There isn't a new set of failures - Ike just noticed that the test is disabled, from when this issue was originally opened. could one of you comment on whether Prashanth's suggestion sounds reasonable?
I didn't parse through the entire issue since a lot of information is stale. Yes David is right, we moved to the logic of filling hostIP of Pod from apiserver to kubelet long time ago. Recording to the comment of Prashanth: Some users might have node with multiple network interfaces, and want to hostIP with specific one by updating Node object.
Fwiw I think we should turn the test on and handle things as they come up.

I think the major source of flake was handled when we did:
With respect to the long-term future of HostIP: I'm in favor of documenting the field as deprecated and asking users to contact us with their use case if they need it (in the API field documentation). As long as the field exists, I agree we should test that it's set correctly.
in this comment you wrote "Ping on this. e2e "Pods should get a host IP [Conformance]" is currently disabled everywhere. Bumping this to P1 and flake." What made you say it is disabled? I looked at Jenkins and it seems to be running it: (root)/Kubernetes%%20suite/PodsshouldgetahostIPConformance/history/ I looked in and didn't see any special handling of [Conformance]. There is an env var CONFORMANCETESTSKIP_REGEX in hack/gingko- but it doesn't seem to be set anywhere. So I don't see anything that should be causing this test to not be run. (As always there's a high likelihood I'm misunderstanding something.)
Sorry, I'm totally blind, the Jenkins history I linked to clearly shows it is being skipped. But I'm still not sure what is causing it to be skipped.
Sorry I wasn't clear; it's being skipped because it was marked pending via . Let's agree to use and labels in the future for clarity's sake.
Also see for use cases of "I need to know my local node" and possible fixes.
I'm going to remove the flake label, since has been merged and that was the suggestion here We can re-add the label if it flakes again. I'm not closing the issue, because we want to deprecate the field longer-term.


Question: ginkgo.It("should be ready immediately after startupProbe succeeds", func() { // Probe workers sleep at Kubelet start for a random time which is at most PeriodSeconds // this test requires both readiness and startup workers running before updating statuses <del> // to avoid flakes, ensure sleep before startup (22s) > readinessProbe.PeriodSeconds cmd := []string{"/bin/sh", "-c", "echo ok >/tmp/health; sleep 22; echo ok >/tmp/startup; sleep 600"} </del> <ins> // to avoid flakes, ensure sleep before startup (32s) > readinessProbe.PeriodSeconds cmd := []string{"/bin/sh", "-c", "echo ok >/tmp/health; sleep 32; echo ok >/tmp/startup; sleep 600"} </ins> readinessProbe := &v1.Probe{ Handler:             execHandler([]string{"/bin/cat", "/tmp/health"}), InitialDelaySeconds: 0, <del> PeriodSeconds:       20, </del> <ins> PeriodSeconds:       30, </ins> } startupProbe := &v1.Probe{ Handler:             execHandler([]string{"/bin/cat", "/tmp/startup"}), InitialDelaySeconds: 0, <del> FailureThreshold:    60, </del> <ins> FailureThreshold:    120, PeriodSeconds:       5, </ins> } p := podClient.Create(startupPodSpec(startupProbe, readinessProbe, nil, cmd))

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Seems to be impacting a few KIND jobs on the master branch the most, but also assorted other jobs see the triage link above Reported by Sample failure: /sig node <!-- Please see the deflaking doc () for more guidance! --
xref probing tests
/assign ref /triage accepted
Thanks Elana, still trying today... Does anyone know if we can use the kubelet logs in e2e tests?
Not very portably in cluster e2e tests at least. Maybe in node_e2e instead.
Have you considered that the issue is not with the test but with the underlying code in kubelet like detailed in this issue Like detailed in the issue I can reliably reproduce (on k8s 1.18) the startup probe passing within a few seconds, but the container is only marked as ready after about a minute.
there is already a fix in the 1.21 release for that (part of a bigger PR): And actually this test was to verify that part of the code, but unfortunately sometimes it doesn't respond as quick as it should (but never on my PC, only in ci/cd tests)...
Some recent flakes:
one of the most common flakes in kind jobs currently failures like: Mar 25 17:03:42.508: Pod became ready in 42.007286524s, more than 5s after startupProbe succeeded. It means that the delay readiness probes were not initiated immediately after startup finished.
Maybe we could speed up the review on
Thank you I will take a look. I'm making a pass through flakes tonight trying to identify if anything is on kind's end
I don't think it's kind... since running on my PC with kind it works 100%, I think the issue is either the resources... or contention in the queue as mentioned to in the comment
/milestone v1.22
commented on PR with the suggested fix
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
Yeah...
Thanks for your great work in keeping things in check
FYI
I have (finally) understood what's happening, and I can even explain why the deflake made things worse... Please have a look at the sleep before the worker starts to process probes: So, you have to remember that we have one worker per pod and per probe, and since we have these probe settings for the test: We have (PeriodSeconds=10) starting sometimes immediately (if the Kubelet has started since more than 10s) or after a quick sleep of rand10s. On the contrary, will likely sleep most of the time for rand60s. This causes a delay in the processing of notifications: If we are lucky, this delay is less than 5s and the test passes: Now you can understand why the deflaked settings were actually worse:
The reason I didn't spot these was that I usually start my cluster once for all and iterate on the tests... so the kubelet had plenty of time to start and I never ended in that sleep.
For the solution, I suggest we either remove the sleep completely, or we find something that would make all 3 probe workers start at the same time to avoid sync issues... For the record, I was able to spot this issue thanks to the following log error (because the channel was overflowing): What do you think ?
I have found another issue in that code: If has already ran (and failed), we go to L245 and don't probe via and therefore wait for the next tick, making the test fail.
This seems consistent with what I found when I was adding e2es for the new probe-level termination grace period and why I did my timing math the way I did.
/reopen I don't think this is fixed, I just hit it again and gubernator is still showing plenty of flakes:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
how to check the exact commit used for each test?

For instance, I am not sure (from the name) that should include my latest fix...
In your PR the fix wasn't there:
you may send cherry-pick to 1.21 for which is using kubelet 1.21, not 1.22.
is using kubelet. Cherry-pick to 1.20 may be needed as well.
I have found this one that failed even with the fix: corresponding to:
I don't know what could explain this atm... but it seems we still have delay propagating container statuses.
In the job artifacts there are multiple places with this info including the clone records/logs, and In testgrid it's a row.
yeah afaict since the tests ran on my PR after your fix was merged into master, it showed the flake was still present :( Prow will always use the latest master when running CI.
Sorry for doubting what you said... I was probably in deny and hoped this annoyance was fixed once for all :(
Plan: we'll see if Clayton's pod lifecycle refactor and/or the jitter refactor for probes affects this.
Clayton's PR jitter refactor
since isn't (likely) coming to 1.22 we could cut off 90% of flakes by increasing to 15s delay:
it seems it fails even after merging so let's continue our discussion on slack
this is not a regression, our suggestion is not block release on it. feel free to remove the milestone
/milestone v1.23
In Kubelet maybe, but in the flakiness of the e2e tests it was. Unfortunately we can't easily explore the current state :/
In the release blocking jobs the frequency is not horrible now, so unblocking is probably fine, it is not as bad as it was. Outside of the release cycle this is still the flakiest test in some jobs, see e.g. where it's the only flake source we have on record currently.
if we end up needing it, we have a timing/period adjustment PR we can take that should help with the flakiness with