Question: discovery "k8s.io/api/discovery/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/apimachinery/pkg/util/version" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/tools/events" "k8s.io/klog/v2"

continuing discussion from and
/sig network /triage accepted
from I think we'll officially declare that we support kernel 5.4+ and nftables command-line 0.9.2+, and we can use a expression to check that. (Which means really we'll support kernel 5.3, but we won't advertise it that way.) FTR (and maybe worth documenting in a comment wherever we put the check): If we decide we need concatenations-with-ranges support again later, that would imply 5.6/0.9.4. If we want to use in sets/maps, that would require at least 0.9.4 (and maybe later depending on exactly how we're using it; we'd have to re-check the release notes). Support for comments on tables/chains/sets/maps/etc (anything but rules and elements) didn't come until 5.10/0.9.7, but knftables itself already does a for that at construct time, and just drops those comments if they're not supported.
There was a and about nft segfaults. It appears that specifically, nft v0.9.4 to v0.9.8 (inclusive) will segfault if asked to do anything when any chain in any table contains a certain kind of set/map as created by nft v1.0.3 or later. Specifically, rules like these trigger it (I think specifically it's caused by using and together, which the anonymous map as-generated-by-1.0.3+ uses. Using a named map rather than an anonymous one does not help.) Note that with nft < 1.0.1, doing any operation requires parsing the entire current ruleset. So that means when kube-proxy is using nft 1.0.3 or newer, nothing else in the host network namespace can use nft 0.9.4 - 0.9.8 for anything_. if the system firewall or something else in the host network namespace is using nft 1.0.3 or newer, then it might create rules that will break kube-proxy if kube-proxy is using nft 0.9.4 - 0.9.8.

If kube-proxy is using nft 1.0.1 or later, then it should not be possible for any other well-behaved component to break it, even if there are future bugs like the current one, because kube-proxy never looks at any tables but its own, so it wouldn't see things that other components did that it couldn't understand. OK, so what does this mean for us? We could just ship the official kube-proxy images with hand-built nft 0.9.2 rather than whatever's current in Debian. Then we guarantee we don't install rules other people can't deal with. Except that then we potentially run into problems if the system uses a much newer nft with some new bug that breaks 0.9.2. (We could do something -esque here, but ... ugh.) I think we actually want to go the opposite way, and have kube-proxy print a warning at startup if your nft is older then 1.0.1, because older versions run the risk of being broken by something someone else is doing in another table. But then there's the problem of us breaking other people. Or, even if we don't break other software, we still make it that admins on some systems would be unable to debug nftables-based problems... I guess we could avoid using the known-problematic syntax... the simplest way to do this would be to revert back to doing endpoints the way that iptables does: of course, that's less efficient, but...
Sounds to me that 1.0.2 is the golden version. It doesn't break 0.9.8, and it doesn't parse the entire current ruleset.
golden relative to the bugs we know about right now anyway... but maybe building our official images with 1.0.2 for a while would be a good plan... (Unfortunately we would have to actually build it ourselves. We can't just pull a 1.0.2 build from a random older Debian release because Debian never shipped that version; bullseye had 0.9.8 and bookworm had 1.0.6.)
thoughts on "use less efficient rules" vs "build images with a very specific nft version"?

build images with nft 1.0.1 seems the best option to me , because it seems that nft < 1.0.1 is not prod ready based on the different behaviors we are observing. If this happens between minors in 1.x.y then I think is a different discussion
The question wasn't just about what version we should use in our images, it was how we should deal with the bad interactions with older versions of nftables. It's one thing to say "nftables kube-proxy will not work when using nft < 1.0.1", but it's another thing to say "you are not allowed to run nftables kube-proxy on hosts where anybody is using nft < 1.0.1 (because otherwise kube-proxy will break other users of nft)". There is no reliable way for us to figure out the version of nft on the host from within the kube-proxy container. The best we could do would be to assume that the nft and kernel versions are roughly contemporary, and use the kernel version as a proxy. nft 1.0.1 was contemporary with kernel 5.16, and nft 1.0.0 was contemporary with 5.13. If we require kernel 5.13 / nft 1.0.1, we can support distros newer than mid 2021: RHEL 9 (kernel 5.14 + backports / nft 1.0.4) Debian 12 / bookworm / newstable (kernel 6.1 / nft 1.0.6) Ubuntu 22.04LTS (kernel 5.15 or newer / nft 1.0.2) (Not sure about SLES?)

Recent Fedora and other non-LTS distros Whereas our current plan of requiring kernel 5.4 / nft 0.9.2 means "newer than late 2019" and adds: RHEL 8 (kernel 4.18 + many backports / originally nft 0.9.3, later 1.0.4) Debian 11 / bullseye / oldstable (kernel 5.10 / nft 0.9.8) Ubuntu 20.04LTS (kernel 5.4 or newer / nft 0.9.3) (Not sure about SLES?) OK, so first decision: do we feel like we need to support running nftables kube-proxy on RHEL 8 / Debian 11 / Ubuntu 20.04LTS, or are we OK with only supporting the newer distros? For comparison, seems to be leaning toward 4.18/4.19 (from late 2018) as the minimum supported kernel for kubernetes as a whole, so requiring kernel 5.13 for nftables would add several more years before we could really even consider making it the default backend for kube-proxy. (We could also have kube-proxy require the newer versions by default, but have a config option to allow the older versions, and document exactly what can go wrong with that.) If we do want to still support the older distros (either unconditionally or via config option), there are two more questions: we decouple our kernel and nft requirements, and say that kube-proxy requires nft = 1.0.1 regardless of what kernel you're using? (I think "yes"?) we want to do some sort of workaround for the segfault bug (either changing the endpoint rules, or shipping the kube-proxy image with nft == 1.0.2 rather than whatever-version-is-currently-in-debian)? (If we had a config option to allow older kernels, we could make that option also control whether we used vmap-style or iptables-style endpoint rules?)

OK, leaning toward: Docs: We document that nftables kube-proxy requires nftables = 1.0.1, and recommends a kernel = 5.13 We also document that, currently, it can be used on systems with kernels as old as 5.4 if you set the option in the kube-proxy config, but (a) this will likely cause problems with other nftables users on the system (with details), and (b) this option is intended to go away at some point as the LTS distros with older kernels become unsupported. Code: We add to We check the nft client version (via a new knftables API), and if it's < 1.0.1, we error out. We check the kernel version (via ) and if it's < 5.13: if is (the default) we error out if is : if an with a expression passes, we continue, after warning that we might break other users on the system if the fails, we error out because the kernel is too oldWe don't change our rules or change the version of nft in our images, because the problems will only affect users who have explicitly opted in.
oh, and CI apparently has a 5.15 kernel, so we won't need to update the kind job config
I envision we'll hit more of these things in the future, so it will be nice you implement it, in a modular way, kind of the iptables runner that has all these different behaviors hidden from the users
The idea was that for things that affect knftables's own functioning, or its public API, it would check those things itself and hide the details from the caller. But the caller would be expected to use to figure out things like "can I use in a rule", since knftables can't just abstract around that (in general). (Right now the only thing knftables abstracts around is whether table/chain/set/map comments are supported; you're always allowed to specify them in the API, but .) I was originally thinking that the "require nft 1.0.1" restriction would be at the kube-proxy level rather than the knftables level, but I'm thinking now it would make more sense to require that for all knftables users.


Question: func TestPodSecurityGAOnly(t *testing.T) { // Disable all alpha and beta features for k, v := range utilfeature.DefaultFeatureGate.DeepCopy().GetAll() { <del> if v.PreRelease == featuregate.Alpha || v.PreRelease == featuregate.Beta { </del> <ins> if k == "AllAlpha" || k == "AllBeta" { // Skip special features. When processed first, special features may // erroneously disable other features. continue } else if v.PreRelease == featuregate.Alpha || v.PreRelease == featuregate.Beta { </ins> defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, k, false)() } }

PR was reverted by because it (erroneously) relied on OpenAPIV3 feature to be enabled, but would non-deterministically leave the feature disabled after running. I tracked it down to the fact that this test disables all or features in a loop which iterates over a , which are known to have unstable iteration order: When key is processed before key , then would disable before s default value is to be saved. Normally, ignores features that have been explicitly set, so it only affects those which fall back on the value. But this issue does not just affect but any non-GA feature. I suggest the loop in be made to skip the special feature keys such as . /cc /cc The value before every feature gate should be restored after running . This test always fails on my machine (just pasted it into ): No response <details</details<details</details<details</details<details</details<details</details<details</details
/assign
Wow, good catch, sorry to make you track down what turned out to be a bug in our test
/sig auth
/area test
/triage accepted


Question: f.NamespacePodSecurityEnforceLevel = admissionapi.LevelPrivileged /* <del> Release: v1.20 Testname: Event resource lifecycle Description: Create an event, the event MUST exist. The event is patched with a new message, the check MUST have the update message. The event is deleted and MUST NOT show up when listing all events. </del> <ins> Release: v1.25 Testname: Event, manage lifecycle of an Event Description: Attempt to create an event which MUST succeed. Attempt to list all namespaces with a label selector which MUST succeed. One list MUST be found. The event is patched with a new message, the check MUST have the update message. The event is updated with a new series of events, the check MUST confirm this update. The event is deleted and MUST NOT show up when listing all events. </ins> */ <del> framework.ConformanceIt("should ensure that an event can be fetched, patched, deleted, and listed", func() { eventTestName := "event-test" ginkgo.By("creating a test event") // create a test event in test namespace _, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).Create(context.TODO(), &v1.Event{ ObjectMeta: metav1.ObjectMeta{ Name: eventTestName, Labels: map[string]string{ "testevent-constant": "true", }, }, Message: "This is a test event", Reason:  "Test", Type:    "Normal", Count:   1, InvolvedObject: v1.ObjectReference{ Namespace: f.Namespace.Name, }, }, metav1.CreateOptions{}) framework.ExpectNoError(err, "failed to create test event") ginkgo.By("listing all events in all namespaces") // get a list of Events in all namespaces to ensure endpoint coverage eventsList, err := f.ClientSet.CoreV1().Events("").List(context.TODO(), metav1.ListOptions{ LabelSelector: "testevent-constant=true", }) framework.ExpectNoError(err, "failed list all events") foundCreatedEvent := false var eventCreatedName string for _, val := range eventsList.Items { if val.ObjectMeta.Name == eventTestName && val.ObjectMeta.Namespace == f.Namespace.Name { foundCreatedEvent = true eventCreatedName = val.ObjectMeta.Name break } } if !foundCreatedEvent { framework.Failf("unable to find test event %s in namespace %s, full list of events is %+v", eventTestName, f.Namespace.Name, eventsList.Items) } ginkgo.By("patching the test event") // patch the event's message eventPatchMessage := "This is a test event - patched" eventPatch, err := json.Marshal(map[string]interface{}{ "message": eventPatchMessage, }) framework.ExpectNoError(err, "failed to marshal the patch JSON payload") _, err = f.ClientSet.CoreV1().Events(f.Namespace.Name).Patch(context.TODO(), eventTestName, types.StrategicMergePatchType, []byte(eventPatch), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch the test event") ginkgo.By("fetching the test event") // get event by name event, err := f.ClientSet.CoreV1().Events(f.Namespace.Name).Get(context.TODO(), eventCreatedName, metav1.GetOptions{}) framework.ExpectNoError(err, "failed to fetch the test event") framework.ExpectEqual(event.Message, eventPatchMessage, "test event message does not match patch message") ginkgo.By("deleting the test event") // delete original event err = f.ClientSet.CoreV1().Events(f.Namespace.Name).Delete(context.TODO(), eventCreatedName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete the test event") ginkgo.By("listing all events in all namespaces") // get a list of Events list namespace eventsList, err = f.ClientSet.CoreV1().Events("").List(context.TODO(), metav1.ListOptions{ LabelSelector: "testevent-constant=true", }) framework.ExpectNoError(err, "fail to list all events") foundCreatedEvent = false for _, val := range eventsList.Items { if val.ObjectMeta.Name == eventTestName && val.ObjectMeta.Namespace == f.Namespace.Name { foundCreatedEvent = true break } } if foundCreatedEvent { framework.Failf("Should not have found test event %s in namespace %s, full list of events %+v", eventTestName, f.Namespace.Name, eventsList.Items) } }) </del> <ins> framework.ConformanceIt("should manage the lifecycle of an event", func() { // As per SIG-Arch meeting 14 July 2022 this e2e test now supersede // e2e test "Event resource lifecycle", which has been removed. </ins> <del> ginkgo.It("should manage the lifecycle of an event", func() { </del> eventTestName := "event-test" ginkgo.By("creating a test event")

[X] test approval issue : [X] test pr : [x] two weeks soak end date : 15 Jul 2022 [x] test promotion pr : According to this APIsnoop query, there are still a single remaining Event endpoint which is untested. replaceCoreV1NamespacedEvent Using a number of existing e2e test practices a new has been created for this pod status endpoint. The e2e logs for this test are listed below. This query shows the event endpoints hit within a short period of running the e2e test. If a test with these calls gets merged, test coverage will go up by 1 point. This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func TestEtcdCreate(t *testing.T) { podA := &api.Pod{ <del> ObjectMeta: api.ObjectMeta{Name: "foo"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: api.NamespaceDefault}, </ins> Status:     api.PodStatus{Host: "machine"}, } podB := &api.Pod{ <del> ObjectMeta: api.ObjectMeta{Name: "foo"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: "foo", Namespace: api.NamespaceDefault}, </ins> Status:     api.PodStatus{Host: "machine2"}, }

The use of AtomicUpdate by genericetcd doesn't handle conflicting writes by rejecting that to a user. It should. Eventually, we may want to hide write conflicts to /pods when status has changed, and to spec when /pods/status changes, but that requires us to have two version counters and that's not worth the effort. Fix incoming.
Also we need a unit/integration test for this.


Question: kube::etcd::install() { ( cd "${KUBE_ROOT}/third_party" <del> curl -fsSL --retry 3 https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz | tar xzf - ln -fns "etcd-v${ETCD_VERSION}-linux-amd64" etcd </del> <ins> if [[ $(uname) == "Darwin" ]]; then download_file="etcd-v${ETCD_VERSION}-darwin-amd64.zip" curl -fsSLO --retry 3 https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/"${download_file}" unzip -o "${download_file}" ln -fns "etcd-v${ETCD_VERSION}-darwin-amd64" etcd rm "${download_file}" else curl -fsSL --retry 3 https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz | tar xzf - ln -fns "etcd-v${ETCD_VERSION}-linux-amd64" etcd fi </ins> kube::log::info "etcd v${ETCD_VERSION} installed. To use:" kube::log::info "export PATH=${PATH}:$(pwd)/etcd" )

is and does not honor the platform it is invoked on. I noticed this when I needed to install etcd on my Mac (which, in turn, was required by and transitively by the pre-commit hooks) and later failed to execute it due to an incompatible binary.
Oh wow, that explains my failures. I ended up using homebrew.


Question: if labels.Set(controller.DesiredState.ReplicaSelector).AsSelector().Empty() { allErrs = append(allErrs, errs.NewInvalid("ReplicationController.ReplicaSelector", controller.DesiredState.ReplicaSelector)) } <ins> selector := labels.Set(controller.DesiredState.ReplicaSelector).AsSelector() labels := labels.Set(controller.DesiredState.PodTemplate.Labels) if !selector.Matches(labels) { allErrs = append(allErrs, errs.NewInvalid("ReplicaController.DesiredState.PodTemplate.Labels", controller.DesiredState.PodTemplate)) } </ins> if controller.DesiredState.Replicas < 0 { allErrs = append(allErrs, errs.NewInvalid("ReplicationController.Replicas", controller.DesiredState.Replicas)) }

If a ReplicaController has a selector that doesn't match the template it'll just create tons and tons of pods. It should validate that the pods it is creating have a good chance of satisfying its selector.
It would also be useful to validate that the selector doesn't obviously overlap with selectors of other replicationControllers.


Question: cd "${KUBE_ROOT}/third_party" if [[ $(uname) == "Darwin" ]]; then download_file="etcd-v${ETCD_VERSION}-darwin-amd64.zip" <del> curl -fsSLO --retry 3 --keepalive-time 2 https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/"${download_file}" </del> <ins> url="https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/${download_file}" kube::util::download_file "${url}" "${download_file}" </ins> unzip -o "${download_file}" ln -fns "etcd-v${ETCD_VERSION}-darwin-amd64" etcd rm "${download_file}" else <del> curl -fsSL --retry 3 --keepalive-time 2 https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz | tar xzf - </del> <ins> url="https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz" download_file="etcd-v${ETCD_VERSION}-linux-amd64.tar.gz" kube::util::download_file "${url}" "${download_file}" tar xzf "${download_file}" </ins> ln -fns "etcd-v${ETCD_VERSION}-linux-amd64" etcd fi kube::log::info "etcd v${ETCD_VERSION} installed. To use:"

Sometimes e2e tests fails during cluster installation because curl fails with: For example see here: https://k8s- Downloading etcd failed. I've run curl to get etcd from github 4400 times. It failed 5 times. So it doesn't happen often.


Question: ## Known Issues - volumeDevices mapping ignored when container is privileged - The `Should recreate evicted statefulset` conformance [test]( https://github.com/kubernetes/kubernetes/blob/master/test/e2e/apps/statefulset.go) fails because `Pod ss-0 expected to be re-created at least once`. This was caused by the `Predicate PodFitsHostPorts failed` scheduling error. The root cause was a host port conflict for port `21017`. This port was in-use as an ephemeral port by another application running on the node. This will be looked at for the 1.18 release. <ins> - client-go discovery clients constructed using `NewDiscoveryClientForConfig` or `NewDiscoveryClientForConfigOrDie` default to rate limits that cause normal discovery request patterns to take several seconds. This is fixed in https://issue.k8s.io/86168 and will be resolved in v1.17.1. As a workaround, the `Burst` value can be adjusted higher in the rest.Config passed into `NewDiscoveryClientForConfig` or `NewDiscoveryClientForConfigOrDie`. </ins> ## Urgent Upgrade Notes ### (No, really, you MUST read this before you upgrade)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With client-go v0.17.0, I observed many "Throttling request" when DiscoveryClient is used to get API group resources from API server. This didn't happen with client-go kubernetes-1.16.0. It looks like originally this kind of discovery requests are not throttled because no rate limiter provided and QPS == 0. But with rate limiter is always set if nil rate limiter is provided. is particularly suspicious because it is always true because DefaultQPS is nonzero. What you expected to happen: Discovery requests should not be throttled as before. How to reproduce it (as minimally and precisely as possible): Create DiscoveryClient and then call with it. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
The NewTokenBucketRateLimiter was via Always negotiate a decoder using ClientNegotiator /cc
To be clear, that was fixing a bug where the documented defaults for rate limiting were not taking effect. I don't think we should completely disable throttling until more of the server-side priority work is completed, but discovery is expected to be bursty, so I could see defaulting to a different amount in the discovery client constructor, similar to
Yeah, everyone must be rate limited. I think we could give discovery a big burst value.
opened


Question: // Now we can start deleting items.  We should use the REST API to ensure that all normal admission runs. // Since we control the endpoints, we know that delete collection works. No need to delete if not established. <del> if apiextensions.IsCRDConditionTrue(crd, apiextensions.Established) { </del> <ins> if OverlappingBuiltInResources()[schema.GroupResource{Group: crd.Spec.Group, Resource: crd.Spec.Names.Plural}] { // Skip deletion, explain why, and proceed to remove the finalizer and delete the CRD apiextensions.SetCRDCondition(crd, apiextensions.CustomResourceDefinitionCondition{ Type:    apiextensions.Terminating, Status:  apiextensions.ConditionFalse, Reason:  "OverlappingBuiltInResource", Message: "instances overlap with built-in resources in storage", }) } else if apiextensions.IsCRDConditionTrue(crd, apiextensions.Established) { </ins> cond, deleteErr := c.deleteInstances(crd) apiextensions.SetCRDCondition(crd, cond) if deleteErr != nil {

What happened: Created a CRD: Created another CRD defining an overlapping type with CustomResourceDefinition: Verify both exist: Deleted the one overlapping with the in-tree CRD type: Verify only the overlapping CRD remains: and is stuck deleting: What you expected to happen: Creation and deletion of the overlapping type CRD would have no effect on the in-tree data. Anything else we need to know?: The only types in the kube-apiserver (currently) that store data under a group prefix are: APIService () CustomResourceDefinition () Custom resources () When a CRD is deleted, all instances of the type it defines are first removed. That means that defining and deleting a CustomResourceDefinition that defines an overlapping APIService or CustomResourceDefinition type will remove existing APIService or CustomResourceDefinition data. We can special-case these two resources in the CRD deletion code, but we need a more robust solution before CRD v1. /cc /sig api-machinery /priority important-soon /area custom-resources
/sub
Is anyone working on this? I'll like to take a stab at this.
Not currently, no. Great. I'd suggest discussion a potential solution here before diving into code. The two main aspects are: -time - Should we forbid creating CRDs that overlap with built-in types? -time - How do we detect a CRD overlapping with a built-in type and skip the etcd cleanup for it
I think we should forbid creating CRDs that overlap (option 1).
That can help, but is not sufficient. Imagine: n forbids creating overlapping CRDs on version n creates MyType, which is allowed because that is not a built-in type upgrades to version n+1 which adds MyType as a built-in type deletes CRD What should happen?
For that, we may need delete-time detection of overlapping types as well (option 2), which may make option 1 redundant. Do you have any advice for detecting overlapping CRDs?
for now, just delete-time detection in


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" utilerrors "k8s.io/apimachinery/pkg/util/errors" <ins> "k8s.io/apimachinery/pkg/util/net" </ins> "k8s.io/apimachinery/pkg/util/strategicpatch" "k8s.io/client-go/kubernetes" <ins> "k8s.io/client-go/util/retry" </ins> corev1helpers "k8s.io/component-helpers/scheduling/corev1" "k8s.io/klog/v2" extenderv1 "k8s.io/kube-scheduler/extender/v1"

If there is a connection failure when updating the Pod status, we don't retry. Furthermore, due to , we won't have another chance until much later. The problem is that if we don't mark the Pod as Unschedulable, other controllers (such as cluster-autoscaler) wouldn't react to these pods. We should have stronger guarantees to mark a Pod as Unschedulable. You need an unschedulable Pod (for example, a Pod with a non matching node affinity) and a flaky connection to apiserver. No response master This is particularly problematic in 1.24, due to . Any <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
That is bad, retrying in place on transient errors should probably be ok.
I'll address it. /assign
Thanks!


Question: // "deleteClaim" to remove the claim from controller's cache and mark bound volume to be released func(ctrl *PersistentVolumeController, reactor *pvtesting.VolumeReactor, test controllerTest) error { // wait until the provision timestamp has been inserted <del> for !ctrl.operationTimestamps.Has("default/claim5-8") { time.Sleep(10 * time.Millisecond) </del> <ins> err := wait.Poll(10*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { return ctrl.operationTimestamps.Has("default/claim5-8"), nil }) if err != nil { return err </ins> } // delete the claim obj := ctrl.claims.List()[0] claim := obj.(*v1.PersistentVolumeClaim) reactor.DeleteClaimEvent(claim) // wait until claim is cleared from cache, i.e., deleteClaim is called <del> for len(ctrl.claims.ListKeys()) > 0 { time.Sleep(10 * time.Millisecond) </del> <ins> err = wait.Poll(10*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { return len(ctrl.claims.ListKeys()) == 0, nil }) if err != nil { return err </ins> } // make sure operation timestamp cache is empty if ctrl.operationTimestamps.Has("default/claim5-8") {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are some places in that wait for conditions . We should update all the places to set something like /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: There are some places in that wait for conditions . We should update all the places to set something like /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /remove-kind bug
I can try to fix this :) /assign
/sig storage


Question: "net" "net/http" "os" <del> "path" </del> "reflect" "regexp" "sort"

go test -v This is because there are a few test cases in dockertools are linux-specific.


Question: } func (e Equalities) deepValueDerive(v1, v2 reflect.Value, visited map[visit]bool, depth int) bool { <ins> defer makeUsefulPanic(v1) </ins> if !v1.IsValid() || !v2.IsValid() { return v1.IsValid() == v2.IsValid() }

This tree has a commit that demonstrates two obviously different objects that should not pass the semantic DeepEqual but do.
What shall we do with this issue for triage? Labels? Priority?
P2 for when he returns , Satnam Singh wrote:
OK I am now sufficiently worried about this. Fixes incoming.


Question: } // Combine all framework configurations. If this results in any duplication, framework // instantiation should fail. <del> var plugins schedulerapi.Plugins plugins.Append(pluginsForPredicates) plugins.Append(pluginsForPriorities) plugins.Append(c.plugins) c.plugins = &plugins </del> <ins> var defaultPlugins schedulerapi.Plugins defaultPlugins.Append(pluginsForPredicates) defaultPlugins.Append(pluginsForPriorities) defaultPlugins.Apply(c.plugins) c.plugins = &defaultPlugins </ins> var pluginConfig []schedulerapi.PluginConfig pluginConfig = append(pluginConfig, pluginConfigForPredicates...)

What would you like to be : Actually disable scheduler plugins according to the disabled list in the scheduler's component config. The semantics are defined here: Part of /sig scheduling /priority important-soon /assign
The following semantics seems vague, and I think we should reach consensus on some cases when disabled plugins are involved. Suppose we have two default plugins, and , the runs before , here are the cases we could meet with the scheduling framework, plugins which run after all the default plugins plugins which should run ahead of the plugins plugin which should run ahead of all the plugins -enable default or custom plugin should return errors
Do you have any other cases not mentioned here?
Those cases sound right, I don't have others.
Is this actually what we want? I don't think so. The comment needs rewording.
I plan to revisit the comments after


Question: "strings" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> v1meta "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/serializer"

The flag has been deprecated for many releases and can be removed. The depreciation guide specifies that admin-only CLI flags can be removed after 6 months: /kind cleanup /sig network /priority backlog
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Thanks In future, please comment/check when taking on an issue, I didn't realize someone was going to do this.
Sorry, I'll comment on the issue and check next time.
That's okay, I only wasted a few minutes on it before I noticed the PR.


Question: // ensureReferencedToken makes sure at least one ServiceAccountToken secret exists, and is included in the serviceAccount's Secrets list func (e *TokensController) ensureReferencedToken(serviceAccount *v1.ServiceAccount) ( /* retry */ bool, error) { <del> if len(serviceAccount.Secrets) > 0 { allSecrets, err := e.listTokenSecrets(serviceAccount) if err != nil { // Don't retry cache lookup errors return false, err } referencedSecrets := getSecretReferences(serviceAccount) for _, secret := range allSecrets { if referencedSecrets.Has(secret.Name) { // A service account token already exists, and is referenced, short-circuit return false, nil } } </del> <ins> if hasToken, err := e.hasReferencedToken(serviceAccount); err != nil { // Don't retry cache lookup errors return false, err } else if hasToken { // A service account token already exists, and is referenced, short-circuit return false, nil </ins> } // We don't want to update the cache's copy of the service account

At least twice now, we've created a namespace and seen that the corresponding service account default secret token is never created. (The service account itself is, but not the default secret.) Namespace creation + apiserver logs logstash-ingester: namespace creation triggered service account creation but no secret logstash-broker: namespace creation triggered service account creation and secret creation Resulting service accounts Resulting secrets Controller-manager logs for broken case Controller-manager logs for working case Version Info:
Dupe of , fixed in
Actually, not a dupe. This may be a consequence of other refactors to controller logic - possibly we are missing / dropping events out of the queue. I would expect it to get fixed on the sync interval though.
We are also seeing this happening from time to time. In our case, it wasn't fixed in the next sync interval but it took actually a little bit more of 24 hours.
can we resolve this and remove it from the 1.6 milestone?
fixed in in 1.4
We just hit this exact issue running 1.5.2. Maybe the fix didn't get the root cause or there was a regression since 1.4? This should be reopened. I waited about an hour to see if the sync interval would resolve this but it did not. I fixed by deleting and recreating the namespace. I can share logs, but they are identical to what the OP noted. Version info:
Working with this seems like it's not fixed in v1.5.2 -- what should we do here? We don't have permission to re-open the ticket, but filing the same ticket with the same logs (just fresher dates) seems passive-aggressive. :-) What's your preference?
no problem, just getting back to this. can reopen this issue.
can you recreate with the controller manager running at and include controller manager log lines from ?
Fresh repo, offending namespace "foo96", with "--v=5" since we already have that logging level in place on my dev stack. This took 96 repros of a "create namespace / monitor for token creation / delete namespace" loop.

Our script's commands, starting just after 16:36:43: Controller manager log lines from The first reference to the offending namespace (foo96) and the one other entry afterwards: All controller manager log lines citing the offending namespace: For comparison, the previous successfully-created namespace: Status as of 20 minutes later:
Also, here's the apiserver logs + controller manager logs, for more action/reaction context: Namespace: foo96 (problem) Namespace: foo95 (success)
that's helpful, thanks. it looks like while the token controller is processing the creation event for the service account, it does a live lookup against etcd and fails to find the service account. what version of etcd are you running with? are you running HA etcd? do you have quorum read enabled?
We run etcd version 2.3.7 (previously saw this on 2.2.x as well). We run etcd HA, with a 3 node cluster. We do not run the apiserver with the --etcd-quorum-read flag enabled.
(Note: My specific logs above from this morning are on etcd 2.2.5, but yeah we've seen repro on 2.3.7 as well.)
yeah, the non-quorum HA etcd config is the issue. I can look into updating the token controller to tolerate that, but I'm positive there are similar issues in many controllers that assume live lookups are reliable
Ahh excellent, glad to have an explanation! So -- now that we know what to google for, we found -- where you and clayton addressed what appears to be this same problem for service accounts, and specifically noted that you wanted to avoid recommending quorum reads. Why is the answer different here? Is it because the downstream impact of non-quorum reads is too diverse to reliably retry every relevant action? (Totally fine if that's the answer, just trying to make sure we understand the reason for what appears to be an entirely opposite conclusion to the same type of problem.)

I still think controllers should be built to tolerate weak reads, and I'm willing to fix the token controller bug here, but three factors make me think this position is a losing battle: quorum read got a lot cheaper with etcd3 there are strong opinions that etcd quorum read should be on () which means that very few people are writing controllers designed to tolerate weak reads all the CI tests I'm aware of run with a single etcd or with quorum reads on
Perfect, works for us. As for fixing this particular token controller bug, I no longer have skin in that game -- your call entirely. :-)


Question: } if npm.podLister != nil { <del> // If the pod is not alive, don't contain it. if _, err := npm.podLister.Pods(pi.Pod.Namespace).Get(pi.Pod.Name); err != nil { klog.V(4).InfoS("Pod doesn't exist in podLister, aborting adding it to the nominator", "pod", klog.KObj(pi.Pod)) </del> <ins> // If the pod was removed or if it was already scheduled, don't nominate it. updatedPod, err := npm.podLister.Pods(pi.Pod.Namespace).Get(pi.Pod.Name) if err != nil { klog.V(4).InfoS("Pod doesn't exist in podLister, aborted adding it to the nominator", "pod", klog.KObj(pi.Pod)) return } if updatedPod.Spec.NodeName != "" { klog.V(4).InfoS("Pod is already scheduled to a node, aborted adding it to the nominator", "pod", klog.KObj(pi.Pod), "node", updatedPod.Spec.NodeName) </ins> return } }

The scheduler cache entered a corrupted state where a Pod, according to the cache dump, was nominated to node A, while the status said it was nominated to node B. This lead to unschedulable low priority pods due to the phantom pod in the nominator cache. The cache to match the Pod status Setup: two nodes with all the resources used. A high priority pod comes in that is able to preempt at least one pod. This might cause these sequence of events X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there. the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Nothing clears this nomination afterwards Usually step 2 happens after 3, so everything works as expected. But in a busy cluster, 3 might be delayed. No response <detailsGKE </details<details</details<details</details<details</details<details</details
/sig scheduling cc
This Pod actually got scheduled: The cache dump was obtained much later at : I think the problem is that we got an update for the Pod (probably the nominated node itself), scheduled it again and assigned a second nominated node for it. The second scheduling attempt failed, but we didn't clear the cache.
/triage accepted
/priority important-soon
Sounds like is related?
It could be the same, or it could be another bug. Note that this is a second scheduling attempt for a Pod that was already scheduled.
My theory of events: X doesn't fit, it gets nominated to and back to scheduling queue. start a second scheduling attempt for Pod X. get a Pod update for the . This puts the pod back into the scheduling queue. second scheduling attempt finishes and schedules the pod successfully in start a third scheduling attempt for Pod X (due to 3) get a Pod update for the . This should have cleared the nominated node in the cache, but there is nothing currently there.

the third scheduling attempt, the pod doesn't fit, so the scheduler nominates it to . Not sure why this pod update fails, but nevertheless, we have a corrupted cache at this point.
thanks for raising up this and the fix. Could you update the "How can we reproduce it (as minimally and precisely as possible)?" section a bit? Per our offline conversation, the issue will leave a "phantom" pod holding a number of resources in the internal nominator cache. And due to that unnecessary resource reservation, low-priority pods may not get scheduled while they should have been. This might be the key trigger of this issue. In practice, it can be caused by a (way-too-quick) movePodBackToActiveQ call (due to relevant/irrelevant events, or backoff's completion) after step 1 and before step 3.
Updated the description. I will update the PRs on Monday.


Question: return errs } uOld, ok := old.(*unstructured.Unstructured) <ins> var oldObject map[string]interface{} </ins> if !ok { <del> uOld = nil // as a safety precaution, continue with validation if uOld self cannot be cast </del> <ins> oldObject = nil } else { oldObject = uOld.Object </ins> } v := obj.GetObjectKind().GroupVersionKind().Version <ins> // ratcheting validation of x-kubernetes-list-type value map and set if newErrs := structurallisttype.ValidateListSetsAndMaps(nil, a.structuralSchemas[v], uNew.Object); len(newErrs) > 0 { if oldErrs := structurallisttype.ValidateListSetsAndMaps(nil, a.structuralSchemas[v], oldObject); len(oldErrs) == 0 { errs = append(errs, newErrs...) } } </ins> // validate x-kubernetes-validations rules if celValidator, ok := a.customResourceStrategy.celValidators[v]; ok { if has, err := hasBlockingErr(errs); has { errs = append(errs, err) } else { <del> err, _ := celValidator.Validate(ctx, nil, a.customResourceStrategy.structuralSchemas[v], uNew.Object, uOld.Object, cel.RuntimeCELCostBudget) </del> <ins> err, _ := celValidator.Validate(ctx, nil, a.customResourceStrategy.structuralSchemas[v], uNew.Object, oldObject, cel.RuntimeCELCostBudget) </ins> errs = append(errs, err...) } }

doesn't do the same embedded resource and x-kubernetes-list-type "map" and "set" invariant validation that happens in custom resource create/update validation. Does that mean embedded objects and list-type fields under custom resource status can contain invalid data? if so, that needs to be fixed in a way that tolerates existing invalid data (ratcheting validation) Originally posted by in /sig api-machinery /kind bug /priority important-longterm
/triage accepted
I would like to collaborate for fixing the validation issue for custom resources and tolerate the existing invalid data. Could you please guide me with some code pointers or files and elaborate on how can I proceed for fixing this bug.
I haven't tested this, but something along the lines of this: we'd need tests to exercise the following scenarios on a CRD that specifies a status subresource with a list property with x-kubernetes-list-type=map and x-kubernetes-list-type=set a custom resource instance that already has duplicate items in the status property list can be updated successfully while still having duplicate items in the list (validation doesn't block an invalid → invalid status update) a custom resource instance that already has duplicate items in the status property list can be updated successfully to set unique items in the list (validation doesn't block an invalid → valid status update) a custom resource instance that does not have duplicate items in the status property list cannot be updated to set duplicate items in the list (validation blocks a valid → invalid status update)
since it's been a couple of months, i'm going to try and take a stab at this /assign
/assign
Hi k8s 1.25 Bug Triage Shadow here, just checking in to see if this is still on track for k8s 1.25? Thank you.
has stalled needing tests and a rebase
Thanks for the update.
Hello with the code freeze approaching, is this still on track for v1.25? Thanks!
not really, the status on is the same (needs tests)
/milestone clear


Question: limits: memory: 300Mi requests: <del> cpu: 100m </del> <ins> cpu: 90m </ins> memory: 200Mi volumeMounts: - name: varlog

This causes one component on master to fail in our standard test clusters. Usually it's NPD: cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-lifecycle


Question: ports[key] = true } <del> // Check for duplicate TargetPort portsPath = specPath.Child("ports") targetPorts := make(map[core.ServicePort]bool) for i, port := range service.Spec.Ports { if (port.TargetPort.Type == intstr.Int && port.TargetPort.IntVal == 0) || (port.TargetPort.Type == intstr.String && port.TargetPort.StrVal == "") { continue } portPath := portsPath.Index(i) key := core.ServicePort{Protocol: port.Protocol, TargetPort: port.TargetPort} _, found := targetPorts[key] if found { allErrs = append(allErrs, field.Duplicate(portPath.Child("targetPort"), port.TargetPort)) } targetPorts[key] = true } </del> // Validate SourceRange field and annotation _, ok := service.Annotations[core.AnnotationLoadBalancerSourceRangesKey] if len(service.Spec.LoadBalancerSourceRanges) > 0 || ok {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig api /sig network What happened: Prior to 1.8 it was possible to specify the same targetPort in the service definition multiple times. In 1.8, validation was that prohibited this. What you expected to happen: I expect backwards compatibility for GA resources, so having the same targetPort multiple times should be allowed. This is a valid use case IMO for target ports. If you take the example below, the load balancer terminates SSL, so that manifest allows the nginx pods to receive http on port 80 while the load balancer serves traffic on 80 & 443. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Original ticket for validation:
cc
/kind bug /area api
Sounds reasonable. WDYT?
Hi, same problem here. Trying to create a service with helm, both 80 and 443 ports using same targetPort, and got this error: $ kubectl version
Feels reasonable, and it seems this is impacting enough people to warrant reverting the original change. WDYT?
Given Service objects are , how a breaking change like this was accepted?
/assign
For the one looking for a temporary way to fix it you have to mix named port with value, see below:
seems to be the PR that shipped this. Why would we want to validate that there are no duplicate target ports anyway? There should be plenty of IP addresses free inside the cluster IP ranges on most clusters (or you have a different problem on your hands) and there are plenty of high-range ports free for the NodePort type. The NAT path through the node should be safe. I'm not sure why this validation is needed. It seems perfectly safe and sane to me. Plus this is a pretty severe breaking change for the ingress-nginx sidecar for pods.


Question: return nil } <del> func (f *FakePodControl) DeletePod(namespace string, podName string) error { </del> <ins> func (f *FakePodControl) DeletePod(namespace string, podID string, object runtime.Object) error { </ins> f.Lock() defer f.Unlock() if f.Err != nil { return f.Err } <del> f.DeletePodName = append(f.DeletePodName, podName) </del> <ins> f.DeletePodName = append(f.DeletePodName, podID) </ins> return nil }

Currently RC emits an event when it's creating a pod: But nothing is emitted when a pod is deleted. We should create event for scaling both ways. cc


Question: setUpCmd    = "setup" tearDownCmd = "teardown" execDir     = "/usr/libexec/kubernetes/kubelet-plugins/net/exec/" <del> X_OK        = 0x1 </del> ) func ProbeNetworkPlugins() []network.NetworkPlugin {

/priority critical-urgent /priority failing-test /kind bug /status approved-for-milestone This test has been failing since 2017-11-22 in which is on the , and prevents us from cutting [v1.9.0-beta.1] (kubernetes/sig-release). Is there work ongoing to bring this job back to green? https://k8s- first bad: https://k8s- this is also failing on (non-release-blocking) triage cluster suspect commit range:
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code freeze, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
/remove-area platform/gke bad copy-pasta
This is a new test that was for a 1.9 alpha feature but actually exposed a bug with a different 1.8 alpha feature. Some discussion in . Since this seems to be a bug introduced in 1.8, I'm not sure if it needs to block 1.9. cc


Question: flags.StringVar(&o.controllerManagerOverridesString, "controllermanager-arg-overrides", "", "comma separated list of federation-controller-manager arguments to override: Example "--arg1=value1,--arg2=value2..."") flags.StringVar(&o.apiServerServiceTypeString, apiserverServiceTypeFlag, string(v1.ServiceTypeLoadBalancer), "The type of service to create for federation API server. Options: 'LoadBalancer' (default), 'NodePort'.") flags.StringVar(&o.apiServerAdvertiseAddress, apiserverAdvertiseAddressFlag, "", "Preferred address to advertise api server nodeport service. Valid only if '"+apiserverServiceTypeFlag+"=NodePort'.") <ins> flags.Int32Var(&o.apiServerNodePortPort, apiserverPortFlag, 0, "Preferred port to use for api server nodeport service (0 for random port assignment). Valid only if '"+apiserverServiceTypeFlag+"=NodePort'.") </ins> flags.BoolVar(&o.apiServerEnableHTTPBasicAuth, "apiserver-enable-basic-auth", false, "Enables HTTP Basic authentication for the federation-apiserver. Defaults to false.") flags.BoolVar(&o.apiServerEnableTokenAuth, "apiserver-enable-token-auth", false, "Enables token authentication for the federation-apiserver. Defaults to false.") }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS using Juju OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ): 4.4.0-59-generic Install tools: Juju Others: What happened: kubefed init entered a infinite loop health checking the federation api service What you expected to happen: kubefed to exit with a success code How to reproduce it (as minimally and precisely as possible): Anything else we need to know: When doing a kubefed init on a cluster setup using VMs on aws (using Juju in my case) the worker nodes have two IPs: one that is visible to the internet and one internal visible only to VMs within aws. In my case for example the VM on aws hosting the kubernetes-worker node had external IP in the 54.174.73.1/24 subnet and it also had an internal IP in the 172.31.60.1/24 subnet. Using a client machine outside aws to bootstrap federation: I got the kubefed client stuck on doing a health check against the clouds internal subnet https://172.31.60.XXX:/healthz . To address this I can use the flag and pass the external IP of any kubernetes-worker node.

With this flag kubeconfig file gets updated with the right endpoint for the federation-apiserver. However, apart from the I should also be able to specify the desired port the service will be listening on. I have to make sure that port is opened on the aws routing/firewall prior to issuing . In the example above, kubefed init will try to reach https://54.174.73.XXX:/healthz, port is randomly assigned and is not reachable. I should be able to first open the port on aws and then do a with and port pinned to .
/cc
/assign
is there a reason why you are not using the default load balanced service on AWS?
I am using Juju to stay cloud agnostic. AWS just happened to be where I deployed this time. Juju just grabs machines (virtual or physical) from wherever you have resources from. In effect your cluster is more like an on-prem than anything else. How would the default load balanced service work?
got it.


Question: storagev1 "k8s.io/api/storage/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <del> "k8s.io/apimachinery/pkg/fields" </del> "k8s.io/apimachinery/pkg/runtime/schema" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/apimachinery/pkg/util/sets"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Based on discussion , events can be throttled and should not be used to validate failure conditions in e2e tests. We should remove the event checks from subpath and local pv tests and just wait for pod failure. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign
could you help look at this for local PV tests?
sure /assign
/area deflake
"Invalid AWS KMS key" also waits for an event

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup tests are flaking 1-2 times a day: https://k8s- Some test logs: https://k8s- https://k8s-
could you take a look?
I'll look into this.
I found why the tests failed, see my analysis of logs: I created a PR () trying to fix the issue. I am still investigating kubelet and trying to reproduce the issue locally.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/lifecycle frozen


Question: // IsClosed checks if the queue is closed func (f *DeltaFIFO) IsClosed() bool { <del> f.closedLock.Lock() defer f.closedLock.Unlock() </del> <ins> f.lock.Lock() defer f.lock.Unlock() </ins> return f.closed }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile:


Question: // readonly paths and masked paths. // This requires the ProcMountType feature flag to be enabled. // +optional <del> ProcMount *ProcMountType `json:"procMount,omitEmpty" protobuf:"bytes,9,opt,name=procMount"` </del> <ins> ProcMount *ProcMountType `json:"procMount,omitempty" protobuf:"bytes,9,opt,name=procMount"` </ins> } type ProcMountType string

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When deploying a Manifest that is compatible with 1.11.3 against a newly provisioned 1.12.0-rc2 cluster, older manifests no longer worked. 2018-10-11T01:00:01.3680048Z error: error validating: error validating data: field [0].securityContext.procMount for v1.SecurityContext is required; if you choose to ignore these errors, turn validation off with --validate=false What you expected to happen: 11.3 compatible manifests should work against How to reproduce it (as minimally and precisely as possible): kind: DaemonSet apiVersion: extensions/v1beta1 metadata: name: traefik-ingress-controller namespace: kube-system labels: k8s-app: traefik-ingress-lb spec: template: metadata: labels: k8s-app: traefik-ingress-lb name: traefik-ingress-lb spec: serviceAccountName: traefik-ingress-controller terminationGracePeriodSeconds: 60 containers: image: traefik name: traefik-ingress-lb ports: name: http containerPort: 80 hostPort: 5001 name: admin containerPort: 8080 securityContext: capabilities: drop: ALL add: NETBINDSERVICE args: --api --kubernetes --logLevel=INFO Anything else we need to know?: Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-06-29T01:07:01Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64" kubectl delivered by VSTS Build. 2018-10-11T00:59:54.7599385Z ##[section]Starting: Deploy Traefik Ingress 2018-10-11T00:59:54.7604992Z ============================================================================== 2018-10-11T00:59:54.7606132Z Task : Deploy to Kubernetes 2018-10-11T00:59:54.7606242Z Description : Deploy, configure, update your Kubernetes cluster in Azure Container Service by running kubectl commands.

2018-10-11T00:59:54.7606322Z Version : 0.1.31 2018-10-11T00:59:54.7606416Z Author : Microsoft Corporation 2018-10-11T00:59:54.7606512Z Help : 2018-10-11T00:59:54.7606590Z ============================================================================== Cloud provider or hardware configuration: ACS-Engine 0.22.4. Windows 1803.
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
My expectation would be that this would default to "securityContext.procMount: default" - This appears to be related to . I notice that you've put this behind a feature gate. I'll admit that I'm relatively new to Kubernetes. My expectation would be that new fields would default - but is there some convention that says for features of a certain size, or of a certain complexity, we default them to on, and then force backwards compatibility via feature disable? It's not my intent to buck standard approach here.
What is it defaulting to for you now? It should default to default.
It's not defaulting at all. In the presence of this older manifest definition: securityContext: capabilities: drop: ALL add: NETBINDSERVICE args: --api --kubernetes --logLevel=INFO It's yelling at me that procMount is required. 2018-10-11T01:00:01.3680048Z error: error validating: error validating data: field [0].securityContext.procMount for v1.SecurityContext is required; if you choose to ignore these errors, turn validation off with --validate=false
ah ok I will do a fix
I'm also getting this error for the with K8s 1.12.

It's not clear from this thread what I need to do to fix this. Can anyone make a suggestion? cc


Question: fs.DurationVar(&s.DeploymentControllerSyncPeriod.Duration, "deployment-controller-sync-period", s.DeploymentControllerSyncPeriod.Duration, "Period for syncing the deployments.") fs.DurationVar(&s.PodEvictionTimeout.Duration, "pod-eviction-timeout", s.PodEvictionTimeout.Duration, "The grace period for deleting pods on failed nodes.") fs.Float32Var(&s.DeletingPodsQps, "deleting-pods-qps", 0.1, "Number of nodes per second on which pods are deleted in case of node failure.") <del> fs.Int32Var(&s.DeletingPodsBurst, "deleting-pods-burst", 10, "Number of nodes on which pods are bursty deleted in case of node failure. For more details look into RateLimiter.") </del> <ins> fs.Int32Var(&s.DeletingPodsBurst, "deleting-pods-burst", 1, "Number of nodes on which pods are bursty deleted in case of node failure. For more details look into RateLimiter.") </ins> fs.Int32Var(&s.RegisterRetryCount, "register-retry-count", s.RegisterRetryCount, ""+ "The number of retries for initial node registration.  Retry interval equals node-sync-period.") fs.MarkDeprecated("register-retry-count", "This flag is currently no-op and will be deleted.")

Is there any good reason to have a nonzero burst rate? Ten is way too high for small clusters, and the number seems to be chosen at random even for large clusters (it's not really the size of any typical failure domain). I think that for now we should set it to zero. Does this sound reasonable to you?
I think that by 0 you mean 1. Goal of was to set it in some sane way. I'll send a simple PR changing it to 1.
Yeah sorry, I meant 1. I'm not suggesting to do the full-blown "make rate proportional to cluster size" thing right now, just to change the burst.
got merged, can this be closed?
Yes, thanks for noticing.
- do you know what description should a PR have to auto close the issue?
I think the problem is that you put a stray period after "Fix"


Question: handler = genericfilters.WithTimeoutForNonLongRunningRequests(handler, c.LongRunningFunc, c.RequestTimeout) handler = genericfilters.WithWaitGroup(handler, c.LongRunningFunc, c.HandlerChainWaitGroup) handler = genericapifilters.WithRequestInfo(handler, c.RequestInfoResolver) <ins> handler = genericapifilters.WithCacheControl(handler) </ins> handler = genericfilters.WithPanicRecovery(handler) return handler }

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: ResourceVersion: resourceVersion, } <del> _, result, err := patcher.Patch(obj.Info.Object, modified, obj.Info.Source, obj.Info.Namespace, obj.Info.Name, nil) </del> <ins> _, result, err := patcher.Patch(obj.Info.Object, modified, obj.Info.Source, obj.Info.Namespace, obj.Info.Name, obj.ErrOut) </ins> return result, err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I tried to with a Deployment spec that was roughly shaped like this (note that the is invalid, since it should be a list of instead of ). This caused kubectl to fail to create a three-way patch and panic. <details<summaryClick here for the stacktrace</summary</detailsWhat you expected to happen: An error is returned How to reproduce it (as minimally and precisely as possible): Create a deployment, then try to diff using the YAML provided above (make sure to replace the name and namespace) Anything else we need to know?: The panic occurs on this line because is nil: It's set to nil here: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig cli
Raised a pr which may fix this bug.


Question: netutils "k8s.io/utils/net" "github.com/onsi/ginkgo" <del> "github.com/onsi/gomega" </del> ) var kubeProxyE2eImage = imageutils.GetE2EImage(imageutils.Agnhost)

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: }) }) <del> // TODO: Enable this when issues are resolved.  Tracked in #21320 //		Context("when scheduling a read only busybox container", func() { //			It("it should return success", func() { //				pod := &api.Pod{ //					ObjectMeta: api.ObjectMeta{ //						Name:      "busybox", //						Namespace: api.NamespaceDefault, //					}, //					Spec: api.PodSpec{ //						// Force the Pod to schedule to the node without a scheduler running //						NodeName: *nodeName, //						// Don't restart the Pod since it is expected to exit //						RestartPolicy: api.RestartPolicyNever, //						Containers: []api.Container{ //							{ //								Image:   "gcr.io/google_containers/busybox", //								Name:    "busybox", //								Command: []string{"sh", "-c", "echo test > /file"}, //								SecurityContext: &api.SecurityContext{ //									ReadOnlyRootFilesystem: &isReadOnly, //								}, //							}, //						}, //					}, //				} //				_, err := cl.Pods(api.NamespaceDefault).Create(pod) //				Expect(err).To(BeNil(), fmt.Sprintf("Error creating Pod %v", err)) //			}) // //			It("it should not write to the root filesystem", func() { //				Eventually(func() string { //					rc, err := cl.Pods(api.NamespaceDefault).GetLogs("busybox", &api.PodLogOptions{}).Stream() //					if err != nil { //						return "" //					} //					defer rc.Close() //					buf := new(bytes.Buffer) //					buf.ReadFrom(rc) //					return buf.String() //				}, time.Second*30, time.Second*4).Should(Equal("sh: can't create /file: Read-only file system")) //			}) // //			It("it should be possible to delete", func() { //				err := cl.Pods(api.NamespaceDefault).Delete("busybox", &api.DeleteOptions{}) //				Expect(err).To(BeNil(), fmt.Sprintf("Error creating Pod %v", err)) //			}) //		}) </del> <ins> Context("when scheduling a read only busybox container", func() { It("it should return success", func() { isReadOnly := true pod := &api.Pod{ ObjectMeta: api.ObjectMeta{ Name:      "busybox", Namespace: api.NamespaceDefault, }, Spec: api.PodSpec{ // Force the Pod to schedule to the node without a scheduler running NodeName: *nodeName, // Don't restart the Pod since it is expected to exit RestartPolicy: api.RestartPolicyNever, Containers: []api.Container{ { Image:   "gcr.io/google_containers/busybox", Name:    "busybox", Command: []string{"sh", "-c", "echo test > /file"}, SecurityContext: &api.SecurityContext{ ReadOnlyRootFilesystem: &isReadOnly, }, }, }, }, } _, err := cl.Pods(api.NamespaceDefault).Create(pod) Expect(err).To(BeNil(), fmt.Sprintf("Error creating Pod %v", err)) }) It("it should not write to the root filesystem", func() { Eventually(func() string { rc, err := cl.Pods(api.NamespaceDefault).GetLogs("busybox", &api.PodLogOptions{}).Stream() if err != nil { return "" } defer rc.Close() buf := new(bytes.Buffer) buf.ReadFrom(rc) return buf.String() }, time.Second*30, time.Second*4).Should(Equal("sh: can't create /file: Read-only file systemn")) }) It("it should be possible to delete", func() { err := cl.Pods(api.NamespaceDefault).Delete("busybox", &api.DeleteOptions{}) Expect(err).To(BeNil(), fmt.Sprintf("Error creating Pod %v", err)) }) }) </ins> }) Describe("metrics api", func() {

logs? link to failed builds? Thanks!
Sorry about that: For context see: Looks like the issue is a compile error that wasn't checked: Looks like this broke the node e2e test suite: 09:00:58 I0216 09:00:58. ] Failure Finished Host e2e-node-coreos-beta.c.kubernetes-jenkins.internal Test Suite Failed to compile e2enode: 09:00:58 09:00:58 # 09:00:58 undefined: isReadOnly
Fixed in


Question: "//test/e2e/framework:go_default_library", "//test/e2e/framework/gpu:go_default_library", "//test/e2e/framework/job:go_default_library", <del> "//test/e2e/framework/kubelet:go_default_library", </del> "//test/e2e/framework/node:go_default_library", "//test/e2e/framework/pod:go_default_library", "//test/e2e/framework/providers/gce:go_default_library",

What happened: In reviewing a test for promotion to conformance I noticed the kubelet API was being accessed via the API server's node proxy endpoint (ref: ). This seemed off, so I reviewed . It doesn't explicitly call out kubelet API access as allowed or disallowed. . and agreed kubelet API access seems like a bad idea. I'm now surveying what we have that does_ assume kubelet API access. It's definitely used by parts of e2e framework and some Conformance tests. I'm trying to verify how critical its use is. It appears that much of it is related to debugging or verifying whether kubelets and the control plane agree on who is running what. What you expected to happen: Find a blanket policy that bans kubelet API access for Conformance tests, whether directly, or indirectly through the API server's node proxy endpoint, at any time. Alternatively, find a policy that says kubelet API access may be allowed, but only for debugging or failure purposes (must not be on the critical path to make the test pass) Alternatively, find a policy that says kubelet API access is allowed, but only indirectly through the API server, and evaluated against the same constraints we use for control plane access (GA, non-optional) If we want to say kubelet API access is banned, we can use this as a tracking issue to demote whichever tests rely on it, and come up with a plan to verify the same behavior without kubelet API access. /area conformance /sig architecture /priority important-soon /milestone v1.16 /assign
Look for anything that ends up calling calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test requires kubelet API access to pass may end up calling it if the test fails, via: Some Conformance tests may end up calling this if the test fails specifically Other areas this is used: only if the test fails required for a pre-stop hook test to pass required to make sure the readonly kubelet apis aren't exposed via api server proxy resource usage tracking tests require kubelet api access to pass tests call this in
Looking for doesn't turn up any additional conformance tests, and I don't feel compelled to link the two other tests that use it as they are unlikely to ever be eligible.

The examples pointed out by on slack, the kubelet's endpoints is required to be available for these to pass
Hello! I'm the bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, I'd like to check its status. The code freeze is starting on August 29th EOD PST (about 5 days from now) and once the code freeze starts, only release-blocking changes will be considered to be merged. Is this issue still planned to be fixed for the 1.16 release?
are there further changes for this targeted at 1.16?
Any update on this?
/milestone v1.17
Bug Triage Release Team checking in. Code freeze for the 1.17 release is on November 14. Will this make it in? Thanks!
/assign
hi is this still targetting for 1.17 or can we move to 1.18? Code freeze is Nov 14th. thanks!
/milestone v1.18
So we've updated the documented policy. My suggestion to close this out is to drop the [Conformance] tag from the tests that currently require the kubelet API to pass, and open issues that they need to be rewritten. Ignore whether the API is needed for diagnostic info in failure modes. SchedulerPredicates [Serial] calls this in BeforeEach, which will happen before some Conformance tests (if this fails, the tests will fail) Conformance test Pods should be submitted and removed [NodeConformance] requires kubelet API access to pass
i'd support this ^^
Hello , Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March. Is this issue still intended for milestone 1.18?
Hello There's been no movement on this issue this year. Should this get pushed to the next milestone?
yes, please John T Skarbek <:
/milestone v1.19
/cc


Question: }, fakeCloud: &fake_cloud.FakeCloud{ Machines: []string{"node0"}, <ins> ExtID: map[string]string{ "node0": "ext-node0", "node1": "ext-node1", }, </ins> }, matchRE:              ".*", expectedRequestCount: 1, // List <del> expectedCreated:      []string{}, </del> <ins> expectedNameCreated:  []string{}, expectedExtIDCreated: []string{}, </ins> expectedDeleted:      []string{}, }, {

I see the following in the kube-controller-manager log: - This is happening to me on HEAD and is the reason why pod status is always showing unknown even when they are scheduled and running.
This is with the vagrant cloud provider. It looks like the delta between the new and old minion is the Node.Spec.ExternalID is being set on the incoming minion but is not set in the originally created minion.
I am not sure how this is not affecting other cloud providers. It looks to me that NodeController populated the node ExternalID here: It looks like the Kubelet provided value is missing the ExternalID
Actually, I am now confused on what the NodeController may or may not be doing here that is causing the status to not get updated.
It looks like the client is passing an ExternalID on the incoming request, but the original value that is in etcd does not have an ExternalID set. Not sure what changed that would have caused this issue to appear.
I think this causes problems: The memory is re-used in a for-loop by Go so referring to a pointer value in a map here will cause problems I think.
Ok, the referenced line is the problem. Sending a PR.
Well, I go back on that now. A simple way to reproduce this is to delete all the minions in your cluster, and verify the node controller is able to add them back. For me, this is causing the node controller to add them back in an unknown status, and then all updates to the node fail. The update fails because the node controller is not setting an ExternalID on the original object.
I am looking into this now. cc/ too
Related issue opened while debugging this:
Debugged through here: I can via manual fmt see that the object has an ExternalID set prior to the call Create. When I look at etcd, I end up with this persisted: Note: it is missing an ExternalID somehow ;-)
Ok, I think I see the issue Nodes returned from here: Do not have an ExternalID Which can make Nodes created from here: End up without an External ID. Having two different nodeClient.Create() paths in this code is confusing. Also we are doing bad things in maps holding pointers to memory allocated in for-loop. We need to fix that ;-)
Yeah, this is bug we should fix. Thanks for looking into this.

If you already have a potential fix, please feel free to grab the issue from me.
I think I have a PR that should fix. Testing that now. Will cc you on the PR.
Thanks for fixing - it did end up affecting AWS as well (AWS isn't in a fully working state just yet, particularly with node discovery).


Question: location              string storageEndpointSuffix string resourceGroup         string <del> cloud                 *Cloud </del> <ins> // store disk URI when disk is in attaching or detaching process diskAttachDetachMap sync.Map cloud               *Cloud </ins> } // getNodeVMSet gets the VMSet interface based on config.VMType and the real virtual machine type.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We found that cloud provider tries to attach a disk, it totally costs 121s in the end one minute(disk still in being detaching state), azure cloud provider tries to delete that disk and it succeeded 121s, attach disk operation succeeded and finally azure vm has attached one non-existing disk What you expected to happen: should not delete an azure disk when that disk is being attached How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: The difficulty here is when a disk is being attached or detached, the disk state is still "unattached", would be the reason why delete disk at that time would succeed: Azure disk RP would have two options to fix this issue: a new disk state: , disk delete operation when disk is being attached or detached Otherwise on azure provider(client) side, we cannot know whether that disk is being attached or detached since the second delete disk operation is done after 1 min Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure


Question: - testname: Jobs, apply changes to status codename: '[sig-apps] Job should apply changes to a job status [Conformance]' description: Attempt to create a running Job which MUST succeed. Attempt to patch <del> the Job status to include a new start time which MUST succeed. An annotation for the job that was patched MUST be found. Attempt to replace the job status with a new start time which MUST succeed. Attempt to read its status sub-resource which MUST succeed </del> <ins> the Job status which MUST succeed. An annotation for the job that was patched MUST be found. Attempt to replace the job status with update which MUST succeed. Attempt to read its status sub-resource which MUST succeed </ins> release: v1.24 file: test/e2e/apps/job.go - testname: Ensure Pods of an Indexed Job get a unique index.

! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign suspecting as cause
Ack, indeed, because this is the suite for alpha features enabled. We have this fix ready for the test itself: /cc
/cc It flakes in
/retitle [Flaking Test] [Conformance] [sig-apps] Job should apply changes to a job status
It used to fail with errors like After it fails with : [FAILED] patched object should have the applied condition with LastTransitionTime time.Date(2024, time.March, 7, 17, 58, 34, 0, time.UTC), got time.Date(2024, time.March, 7, 17, 58, 33, 0, time.Local) instead In [It] at: @ 03/07/24 17:58:34.109 Here I open the follow up fix: PTAL.


Question: "Rev": "787624de3eb7bd915c329cba748687a3b22666a6" }, { <ins> "ImportPath": "github.com/grpc-ecosystem/go-grpc-prometheus", "Rev": "2500245aa6110c562d17020fb31a2c133d737799" }, { </ins> "ImportPath": "github.com/hashicorp/golang-lru", "Rev": "a0d98a5f288019575c6d1f4bb1573fef2d1fcdc4" },

/kind feature Can we provide a way to enable etcd clientv3 prometheus metrics at the apiserver side ? The etcd.clientv3 interface is instantiated and it does not enable metrics. In file in coreos/etcd, one can see the to pass to enable metrics. It would be great if the apiserver has a command line parameter, config/environment variable to enable etcd.clientv3 metrics.
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is quite similar to
Yes, this is a good idea, in fact we just released a new version of the package required for this.
/assign
I can do it.
oh, maybe I should use the new features to implement, I will continue to modify it.
are you referring to injecting the registry to use? Yes I would very much appreciate that, even if we (for now) inject the , that will allow us to modify that more easily in the future.


Question: oldPod := pods[0].(*v1.Pod) newPod := oldPod.DeepCopy() <del> if podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be unchanged for copied pod") </del> <ins> if podChanged(oldPod, newPod) { t.Errorf("Expected pod to be unchanged for copied pod") </ins> } newPod.Spec.NodeName = "changed" <del> if !podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be changed for pod with NodeName changed") </del> <ins> if !podChanged(oldPod, newPod) { t.Errorf("Expected pod to be changed for pod with NodeName changed") </ins> } newPod.Spec.NodeName = oldPod.Spec.NodeName newPod.ObjectMeta.ResourceVersion = "changed" <del> if podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be unchanged for pod with only ResourceVersion changed") </del> <ins> if podChanged(oldPod, newPod) { t.Errorf("Expected pod to be unchanged for pod with only ResourceVersion changed") </ins> } newPod.ObjectMeta.ResourceVersion = oldPod.ObjectMeta.ResourceVersion newPod.Status.PodIP = "1.2.3.1" <del> if !podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be changed with pod IP address change") </del> <ins> if !podChanged(oldPod, newPod) { t.Errorf("Expected pod to be changed with pod IP address change") </ins> } newPod.Status.PodIP = oldPod.Status.PodIP newPod.ObjectMeta.Name = "wrong-name" <del> if !podAddressChanged(oldPod, newPod) { t.Errorf("Expected address to be changed with pod name change") </del> <ins> if !podChanged(oldPod, newPod) { t.Errorf("Expected pod to be changed with pod name change") </ins> } newPod.ObjectMeta.Name = oldPod.ObjectMeta.Name <ins> saveConditions := oldPod.Status.Conditions oldPod.Status.Conditions = nil if !podChanged(oldPod, newPod) { t.Errorf("Expected pod to be changed with pod readiness change") } oldPod.Status.Conditions = saveConditions </ins> } func TestDetermineNeededServiceUpdates(t *testing.T) {

This test has started failing on our e2e test suite, erroring out with "Failed to get endpoints..." https://k8s- https://k8s- Example failures https://k8s- https://k8s- cc cc
This may be related to our recently-merged endpoints change in . I'll look into it.
I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign
GitHub didn't allow me to assign the following users: joelsmith. Note that only can be assigned. <detailsIn response to : I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> = vendor/go.opencensus.io licensed under: = Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!)  The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. = vendor/go.opencensus.io/LICENSE 175792518e4ac015ab6696d16c4f607e </ins>

What happened: Running manually includes this file in LICENSES. Running complains if it is present. The file was incorrectly dropped in This needs to be resolved before 1.19 ships /priority important-soon /milestone v1.19 /cc
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
fixed in