<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The endpointSliceTracker creates a set of resource versions for each service, the resource versions in the set could be deleted when endpointslices are deleted, but the set and its key in the map is never deleted, leading to memory leak. What you expected to happen: The set of resource versions should be delete from endpointSliceTracker when the service is deleted. Environment: Kubernetes version (use ): 1.18.5
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: logger.V(4).Info(log("calling NodeUnprepareResource rpc"), "request", req) defer logger.V(4).Info(log("done calling NodeUnprepareResources rpc"), "response", resp, "err", err) <del> if r.nodeClientCreator == nil { return nil, errors.New("failed to call NodeUnprepareResources. nodeClientCreator is nil") } nodeClient, nodeClientOld, closer, err := r.nodeClientCreator(r.addr) </del> <ins> conn, err := r.plugin.getOrCreateGRPCConn() </ins> if err != nil { return nil, err } <del> defer closer.Close() </del> <ins> nodeClient := drapb.NewNodeClient(conn) nodeClientOld := drapbv1alpha2.NewNodeClient(conn) </ins> ctx, cancel := context.WithTimeout(ctx, PluginClientTimeout) defer cancel()

See comment for details:
/sig node
/area kubelet
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/triage accepted


Question: func NewCmdDescribe(f cmdutil.Factory, out, cmdErr io.Writer) *cobra.Command { options := &resource.FilenameOptions{} <del> describerSettings := &printers.DescriberSettings{} </del> <ins> describerSettings := &printers.DescriberSettings{ ShowEvents: true, } </ins> // TODO: this should come from the factory, and may need to be loaded from the server, and so is probably //   going to have to be removed


Question: assert.NoError(t, err, test.description) assert.Equal(t, test.expected, real.FailureDomain, test.description) <ins> assert.Equal(t, strings.ToLower(cloud.Location), real.Region, test.description) </ins> } }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Azure cloud provider gets the availability zone from IMDS directly and the zone is usually in lower cases in most regions. But there're chances this is not true in some regions, hence customer's workloads depend on availability zone may not work. What you expected to happen: The availability zone is always in lower cases. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure
/assign
Fixed by .


Question: framework.Failf("Pod should have been deleted or have DeletionTimestamp, but instead got: %s", retrievedPod) } }) <ins> ginkgo.It("should run through the lifecycle of a CSIDriver", func(ctx context.Context) { // Create client client := f.ClientSet.StorageV1().CSIDrivers() defaultFSGroupPolicy := storagev1.ReadWriteOnceWithFSTypeFSGroupPolicy csiDriverLabel := map[string]string{"e2e-test": f.UniqueName} csiDriverLabelSelector := labels.SelectorFromSet(csiDriverLabel).String() // Driver that supports only Ephemeral driver1 := &storagev1.CSIDriver{ ObjectMeta: metav1.ObjectMeta{ Name:   "inline-driver-" + string(uuid.NewUUID()), Labels: csiDriverLabel, }, Spec: storagev1.CSIDriverSpec{ VolumeLifecycleModes: []storagev1.VolumeLifecycleMode{storagev1.VolumeLifecycleEphemeral}, FSGroupPolicy:        &defaultFSGroupPolicy, }, } // Driver that supports both Ephemeral and Persistent driver2 := &storagev1.CSIDriver{ ObjectMeta: metav1.ObjectMeta{ Name:   "inline-driver-" + string(uuid.NewUUID()), Labels: csiDriverLabel, }, Spec: storagev1.CSIDriverSpec{ VolumeLifecycleModes: []storagev1.VolumeLifecycleMode{ storagev1.VolumeLifecyclePersistent, storagev1.VolumeLifecycleEphemeral, }, FSGroupPolicy: &defaultFSGroupPolicy, }, } ginkgo.By("Creating two CSIDrivers") createdDriver1, err := client.Create(ctx, driver1, metav1.CreateOptions{}) framework.ExpectNoError(err, "Failed to create first CSIDriver") createdDriver2, err := client.Create(ctx, driver2, metav1.CreateOptions{}) framework.ExpectNoError(err, "Failed to create second CSIDriver") _, err = client.Create(ctx, driver1, metav1.CreateOptions{}) if !apierrors.IsAlreadyExists(err) { framework.Failf("expected 409, got %#v", err) } ginkgo.By(fmt.Sprintf("Getting %q & %q", createdDriver1.Name, createdDriver2.Name)) retrievedDriver1, err := client.Get(ctx, createdDriver1.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get CSIDriver %q", createdDriver1.Name) gomega.Expect(retrievedDriver1.UID).To(gomega.Equal(createdDriver1.UID)) retrievedDriver2, err := client.Get(ctx, createdDriver2.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Failed to get CSIDriver %q", createdDriver2.Name) gomega.Expect(retrievedDriver2.UID).To(gomega.Equal(createdDriver2.UID)) ginkgo.By(fmt.Sprintf("Patching the CSIDriver %q", createdDriver2.Name)) payload := "{"metadata":{"labels":{"" + createdDriver2.Name + "":"patched"}}}" patchedCSIDriver, err := client.Patch(ctx, createdDriver2.Name, types.StrategicMergePatchType, []byte(payload), metav1.PatchOptions{}) framework.ExpectNoError(err, "failed to patch CSIDriver %q", createdDriver2.Name) gomega.Expect(patchedCSIDriver.Labels[createdDriver2.Name]).To(gomega.ContainSubstring("patched"), "Checking that patched label has been applied") ginkgo.By(fmt.Sprintf("Updating the CSIDriver %q", createdDriver2.Name)) var updatedCSIDriver *storagev1.CSIDriver err = retry.RetryOnConflict(retry.DefaultRetry, func() error { csiDriver, err := client.Get(ctx, createdDriver2.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get CSIDriver %q", createdDriver2.Name) csiDriver.Labels[retrievedDriver2.Name] = "updated" updatedCSIDriver, err = client.Update(ctx, csiDriver, metav1.UpdateOptions{}) return err }) framework.ExpectNoError(err, "failed to update CSIDriver %q", createdDriver2.Name) gomega.Expect(updatedCSIDriver.Labels[createdDriver2.Name]).To(gomega.ContainSubstring("updated"), "Checking that updated label has been applied") ginkgo.By(fmt.Sprintf("Listing all CSIDrivers with the labelSelector: %q", csiDriverLabelSelector)) driverList, err := client.List(ctx, metav1.ListOptions{LabelSelector: csiDriverLabelSelector}) framework.ExpectNoError(err, "Failed to list all CSIDrivers with the labelSelector %q", csiDriverLabelSelector) gomega.Expect(driverList.Items).To(gomega.HaveLen(2), "filtered list should have 2 items, got: %s", driverList) ginkgo.By(fmt.Sprintf("Deleting CSIDriver %q", createdDriver1.Name)) err = client.Delete(ctx, createdDriver1.Name, metav1.DeleteOptions{}) framework.ExpectNoError(err, "Failed to delete CSIDriver %q", createdDriver1.Name) ginkgo.By(fmt.Sprintf("Confirm deletion of CSIDriver %q", createdDriver1.Name)) retrievedDriver, err := client.Get(ctx, createdDriver1.Name, metav1.GetOptions{}) switch { case apierrors.IsNotFound(err): // Okay, normal case. case err != nil: framework.Failf("expected 404, got %#v", err) case retrievedDriver.DeletionTimestamp != nil: // Okay, normal case. default: framework.Failf("CSIDriver should have been deleted or have DeletionTimestamp, but instead got: %s", retrievedDriver) } ginkgo.By(fmt.Sprintf("Deleting CSIDriver %q via DeleteCollection", createdDriver2.Name)) err = client.DeleteCollection(ctx, metav1.DeleteOptions{}, metav1.ListOptions{LabelSelector: createdDriver2.Name + "=updated"}) framework.ExpectNoError(err, "Failed to delete CSIDriver %q", createdDriver2.Name) ginkgo.By(fmt.Sprintf("Confirm deletion of CSIDriver %q", createdDriver2.Name)) retrievedDriver, err = client.Get(ctx, createdDriver2.Name, metav1.GetOptions{}) switch { case apierrors.IsNotFound(err): // Okay, normal case. case err != nil: framework.Failf("expected 404, got %#v", err) case retrievedDriver.DeletionTimestamp != nil: // Okay, normal case. default: framework.Failf("CSIDriver should have been deleted or have DeletionTimestamp, but instead got: %s", retrievedDriver) } }) </ins> })

[X] test approval issue : [X] test pr : 26 May 2023 [x] two weeks soak end date : 9 June 2023 [x] test promotion pr : [x] remove endpoint from [x] End of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. According to following APIsnoop query, there are still 3 CSIDriver endpoints that are not tested for Conformance. Using a number of existing e2e test practices a new has been created to address these 3 endpoints. The e2e logs for this test are listed below. This query shows the following csiDriver endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection.
Reopened this issue. <detailsIn response to : /reopen For end of release 1.28 Release: Remove duplicate Conformance test the do not cover Patch, Replace, DeleteCollection. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package state import ( "errors" "fmt" "sync" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager" checkpointerrors "k8s.io/kubernetes/pkg/kubelet/checkpointmanager/errors" ) type Checkpointer interface { GetOrCreate() (*Checkpoint, error) Store(*Checkpoint) error } type checkpointer struct { sync.RWMutex checkpointManager checkpointmanager.CheckpointManager checkpointName    string } // NewCheckpointer creates new checkpointer for keeping track of claim info  with checkpoint backend func NewCheckpointer(stateDir, checkpointName string) (Checkpointer, error) { if len(checkpointName) == 0 { return nil, fmt.Errorf("received empty string instead of checkpointName") } checkpointManager, err := checkpointmanager.NewCheckpointManager(stateDir) if err != nil { return nil, fmt.Errorf("failed to initialize checkpoint manager: %w", err) } checkpointer := &checkpointer{ checkpointManager: checkpointManager, checkpointName:    checkpointName, } return checkpointer, nil } // GetOrCreate gets list of claim info states from a checkpoint // or creates empty list if checkpoint doesn't exist func (sc *checkpointer) GetOrCreate() (*Checkpoint, error) { sc.Lock() defer sc.Unlock() checkpoint, err := NewCheckpoint(nil) if err != nil { return nil, fmt.Errorf("failed to create new checkpoint: %w", err) } err = sc.checkpointManager.GetCheckpoint(sc.checkpointName, checkpoint) if errors.Is(err, checkpointerrors.ErrCheckpointNotFound) { err = sc.store(checkpoint) if err != nil { return nil, fmt.Errorf("failed to store checkpoint %v: %w", sc.checkpointName, err) } return checkpoint, nil } if err != nil { return nil, fmt.Errorf("failed to get checkpoint %v: %w", sc.checkpointName, err) } return checkpoint, nil } // Store stores checkpoint to the file func (sc *checkpointer) Store(checkpoint *Checkpoint) error { sc.Lock() defer sc.Unlock() return sc.store(checkpoint) } // store saves state to a checkpoint, caller is responsible for locking func (sc *checkpointer) store(checkpoint *Checkpoint) error { if err := sc.checkpointManager.CreateCheckpoint(sc.checkpointName, checkpoint); err != nil { return fmt.Errorf("could not save checkpoint %s: %w", sc.checkpointName, err) } return nil } </ins>

When adding the field to , old checkpoint files became invalid because the checksum that gets calculated by hashing the output in no longer matches. This caused kubelet to fail to start with: Here's what the checksum in a unit test gets calculcated for: Note the new . Not sure what the right behavior is. Perhaps kubelet should have ignored the invalid checkpoint? Revert the update in The state.DRAManagerCheckpoint contains a version field, which is set to v1.I think that's where things fall apart: that version must exactly match the type that gets serialized, which is not the case here. The type changes over time, but the version is fixed. Let's reconsider what should get checkpointed and how we can make that stable. Also, is checkpointing really worth it? What bad things happen if we remove it? master n/a No response No response No response /sig node /cc
/cc
We originally didn't have checkpointing and it was by in to fix
The reasons for checkpoint were: (what Kevin mentioned) of the pod resource API DAR extensions concerns raised on what happen if kubelet restart and we get GetPodResourcesRequest. The request should return the DRA claimed devices but if we didn't checkpoint the DRA cache is empty and we would return empty resources force delete see [2] My original PR didn't checkpoint the ResourceHandle is was in PR [2]. Maybe we should exclude ResourceHandle from the checksum hash calculation [1] [2] -
So the cache makes sense. What doesn't make sense to me is how the consistency of that on-disk file is checked and how it is versioned. Shouldn't the checksum be calculated for the bytes on disk (version-independent) instead of the in-memory representation (version-dependent)? Or is the goal to protect against "I can't read this data anymore" instead of "bit has flipped on disk"? The assumption is that the content can always be decoded from what was written in the past into the type which is used now. This works with JSON if the newer type is a true superset of the old one, but that's not necessarily guaranteed. The proper way of doing this is to define a scheme which includes conversion between different versions.

This cannot be done when embedding API types from because conversion between current v1alpha2 and a future v1beta1 can only be done in the apiserver, not client-side. I don't see a quick solution here. In the short-term, how do we deal with version upgrades? Require that users drain the node (thus clearing the cache) before upgrading or downgrading kubelet?
I am still trying to understand the problem. the problem that we checksum the version? the problem that we don't have conversion mechanism to move from version X -version Y on upgrade? the issue related only for upgrade or there are other cases which checksum changes? In general we the version to checkpoint file so that in the future when the DRA checkpoint struct changed we can have mechanism to convert old checkpoint version to new checkpoint version (in the initial PR we didn't implement conversion mechanism as I thought it will be on DRA checkpoint struct change) . If you take a look on the CPU Manager checkpoint [1] you will see that it detecting if the scheme matches to V1 first and if not it assume V2. We wanted that it will part of the checkpoint file so we won't need such detection and in the file we will know that correct version of the DRA checkpoint struct. Now if someone change the DRA checkpoint scheme and didn't implement migration flow (similar to CPU manager [2]) or change the version that indeed an issue, [1] - [2] -
The problem is that after an upgrade of kubelet from <1.30 to 1.30, it won't be able to read an existing, non-empty checkpoint file. Worse, it will refuse to start because of that. The root cause is the lack of proper versioning and lack of conversion code. Any version change potentially has this problem (including downgrades). I'm not aware of other failure scenarios.
Since it is not a GA feature, we cannot fix this in 1.30. But marking it as important long-term. /priority important-longterm
This has to be solved before graduating from alpha
I intend to move all version-dependent code into drivers (). This will simplify the checkpoint content a bit, but the overall questions around its design remain.

/kind feature /remove-kind bug /triage accepted I think a good forum to discuss this could be the KEP for DRA, though we can keep this tracker here
Isn't it because checkpoint structure was changed without incrementing version and adding conversion code? Anyway, I'd propose to remove version from the checkpoint structure and comment it to prevent changing it without changing its version. the implementation should be similar to the cpu maanager checkpointing. If it's been working for CPU (and device?) managers for some time, it should hopefully work for DRA as well. Any other ideas?
Yes. I haven't looked into this again. Can you implement your proposal on top of and submit a PR for review, or do you want to discuss this first?
if you're ok with my proposal I'm ok to implement it. Otherwise it needs to be discussed further.
I don't fully understand the implications without looking into this further, so I can say neither yes nor no to it at this point. do you have an opinion?


Question: return } <del> podChanged := podAddressChanged(oldPod, newPod) </del> <ins> podChangedFlag := podChanged(oldPod, newPod) </ins> // Check if the pod labels have changed, indicating a possibe // change in the service membership

This test has started failing on our e2e test suite, erroring out with "Failed to get endpoints..." https://k8s- https://k8s- Example failures https://k8s- https://k8s- cc cc
This may be related to our recently-merged endpoints change in . I'll look into it.
I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign
GitHub didn't allow me to assign the following users: joelsmith. Note that only can be assigned. <detailsIn response to : I verified that the flakiness introduced was a result of . A fix in should be merging soon. /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // underlying storage will be destroyed on this servers shutdown. func (s *GenericAPIServer) InstallAPIGroups(apiGroupInfos ...*APIGroupInfo) error { for _, apiGroupInfo := range apiGroupInfos { <ins> if len(apiGroupInfo.PrioritizedVersions) == 0 { return fmt.Errorf("no version priority set for %#v", *apiGroupInfo) } </ins> // Do not register empty group or empty version.  Doing so claims /apis/ for the wrong entity to be returned. // Catching these here places the error  much closer to its origin if len(apiGroupInfo.PrioritizedVersions[0].Group) == 0 {

I met a panic in apiserver package when I create an aggregated API. not panic Do not call No response <details</details<details</details<details</details<details</details<details</details<details</details
Reference PR to fix:
/sig api-machinery
/cc /triage accepted


Question: errCh := make(chan error) go func() { <del> conn, err := sctp.DialSCTP("sctp", nil, addr) if err == nil { conn.Close() </del> <ins> conn, err = sctp.DialSCTP("sctp", nil, addr) if err != nil { errCh <- err return } defer func() { errCh <- conn.Close() }() if _, err = conn.Write([]byte(fmt.Sprintf("%sn", data))); err != nil { errCh <- err return } if _, err = conn.Read(buf); err != nil { errCh <- err return </ins> } <del> errCh <- err </del> }() select {

How to repro: an agnhost pod listening on sctp into the pod and use the connect command to probe against the same pod The stack trace that points to and this is from the vendored library , /kind bug /sig network /help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Run an agnhost pod listening on sctpExec into the pod and use the connect command to probe against the same pod The stack trace that points to /sig network /help </details
/assign
/triage accepted
here we are immediately closing the connection after successful connect leading to EOF at server end (it is expecting data) causing the failure. We can tweak this to connect send some data just like UDP and then exit? Works locally writing "hostname" to the connection. [EDIT] For me it sometimes failed with EOF, sometime with nil pointer exception at . This reads sock opts via syscall and we immediately close the connection .. kind of a race between the client closing connection and server reading fds?
the panic seems a bug in that project, no? Also SCTP, has a establishment handshake, so the data should not be relevant in this case
returns fd and socket address, for is used to implement (always read from struct field). The SCTP implementation discards the returned socket address by call. , and is always a new syscall (). In our case the client immediately closes the connection, there can be chances where SCTPGETPEERADDRS will fail and will return nil in these cases. I think this is reasonable for their project. We call and panic is raised by calling on nil object. We can easily avoid this by not null check.

Also, I strongly believe that the client handling should be done in a separate go-routine instead of the main loop to avoid errors like this crashing the pod. [EDIT] The client in our case needs to send some data after connection, or else the server will throw EOF and we currently have assertions that use for errors.
panicing does not seem very reasonable It looks to me that the problem is that the SCTPConn struct needs to check if the connection is still valid before trying to access the file descriptor , see the std library implementation and then, all methods implement this check to me it is a bug on the implementation
The implementation is not causing any panic, it simply returns nil. Panic is caused when we try to call on nil object.
oh, sorry, my bad I should read your comment with more detail, is well explained there I recognize I have zero idea of how sctp works internally, maybe we can just be more resilient on our tests and code and detect if is nil and avoid getting into the complexity of this protocol ... sending some random data also sounds good, we do this for udp IIRC
are you able to send a PR to add those improvements to the test image?


Question: # Kubernetes API and Release Versioning <ins> Reference: [Semantic Versioning](http://semver.org) </ins> Legend: <del> * **Kube X.Y.Z** refers to the version of Kubernetes that is released. This versions all components: apiserver, kubelet, kubectl, etc.  (**X** is the major version, **Y** is the minor version, and **Z** is the patch version.) </del> <ins> * **Kube X.Y.Z** refers to the version (git tag) of Kubernetes that is released. This versions all components: apiserver, kubelet, kubectl, etc.  (**X** is the major version, **Y** is the minor version, and **Z** is the patch version.) </ins> * **API vX[betaY]** refers to the version of the HTTP API. ## Release versioning ### Minor version scheme and timeline <del> * Kube X.Y.0-alpha.W, W > 0: Alpha releases are released roughly every two weeks directly from the master branch. No cherrypick releases. If there is a critical bugfix, a new release from master can be created ahead of schedule. * Kube X.Y.Z-beta.W: When master is feature-complete for Kube X.Y, we will cut the release-X.Y branch 2 weeks prior to the desired X.Y.0 date and cherrypick only PRs essential to X.Y.  This cut will be marked as X.Y.0-beta.0, and master will be revved to X.Y+1.0-alpha.0.  If we're not satisfied with X.Y.0-beta.0, we'll release other beta releases, (X.Y.0-beta.W | W > 0) as necessary. * Kube X.Y.0: Final release, cut from the release-X.Y branch cut two weeks prior.  X.Y.1-beta.0 will be tagged at the same commit on the same branch.  X.Y.0 occur 3 to 4 months after X.Y-1.0. * Kube X.Y.Z, Z > 0: [Patch releases](#patch-releases) are released as we cherrypick commits into the release-X.Y branch, (which is at X.Y.Z-beta.W,) as needed.  X.Y.Z is cut straight from the release-X.Y branch, and X.Y.Z+1-beta.0 is tagged on the same commit. </del> <ins> * Kube X.Y.0-alpha.W, W > 0 (Branch: master) * Alpha releases are released roughly every two weeks directly from the master branch. * No cherrypick releases. If there is a critical bugfix, a new release from master can be created ahead of schedule. * Kube X.Y.Z-beta.W (Branch: release-X.Y) * When master is feature-complete for Kube X.Y, we will cut the release-X.Y branch 2 weeks prior to the desired X.Y.0 date and cherrypick only PRs essential to X.Y. * This cut will be marked as X.Y.0-beta.0, and master will be revved to X.Y+1.0-alpha.0. * If we're not satisfied with X.Y.0-beta.0, we'll release other beta releases, (X.Y.0-beta.W | W > 0) as necessary. * Kube X.Y.0 (Branch: release-X.Y) * Final release, cut from the release-X.Y branch cut two weeks prior. * X.Y.1-beta.0 will be tagged at the same commit on the same branch. * X.Y.0 occur 3 to 4 months after X.(Y-1).0. * Kube X.Y.Z, Z > 0 (Branch: release-X.Y) * [Patch releases](#patch-releases) are released as we cherrypick commits into the release-X.Y branch, (which is at X.Y.Z-beta.W,) as needed. * X.Y.Z is cut straight from the release-X.Y branch, and X.Y.Z+1-beta.0 is tagged on the followup commit that updates pkg/version/base.go with the beta version. * Kube X.Y.Z, Z > 0 (Branch: release-X.Y.Z) * These are special and different in that the X.Y.Z tag is branched to isolate the emergency/critical fix from all other changes that have landed on the release branch since the previous tag * Cut release-X.Y.Z branch to hold the isolated patch release * Tag release-X.Y.Z branch + fixes with X.Y.(Z+1) * Branched [patch releases](#patch-releases) are rarely needed but used for emergency/critical fixes to the latest release * See [#19849](https://issues.k8s.io/19849) tracking the work that is needed for this kind of release to be possible. </ins> ### Major version timeline

This is a top-level/meta issue that may spawn sub-issues. The doc has been recently updated to cover a branching a branch scenario. This is a rare or unlikely occurrence (depending on the teams velocity/needs), but important to plan for nonetheless. Refer to that doc for more details. Any assumptions of a X.Y.Z format need to be dealt with in k8s tooling as well as GKE should we ever want/need to implement this.
It should also be noted that should we need to go this route, we will want to switch to a 4 part version number to avoid any ambiguity (and support hassles) of a mix of 3 and 4 part versions.
The best route here to avoid modifications to the semantic versioning scheme we are currently using for K8S, is to update the tooling to handle branching from a X.Y.Z tag, but maintain the 3-part version scheme. The doc will be updated with details.
Release tooling for branching and building complete in pending TODO list for a zero-day: branch of X.Y.Z (45 minutes) of new branch Do we need a complete set of test infrastructure here or can we define some subset for this purpose? Do we have any mechanisms for easily/quickly spinning up a set of testing infrastructure. Whether we do or not, what is the required setup time for something like this? and submit queue wait (up to N hours) test time for new branch + patch (N hours) the final release from new branch (45 minutes)
Thanks Internal to Google, the time from a PR to a built binary that could potentially be rolled out is I think a bit shorter than the TODO list you mentioned above. Would it make sense to start from e.g. a 1 hour PR-to-rollout target and see what we'd have to do to get there?
yes. For the sake of consistency the initial branch follows the current model of creating the tags and initial builds (and pushes). We could short-circuit that process to just branch and tag which would reduce the time significantly, closer to a few minutes. Then the actual release with a build and push will take around 45 minutes.

The more variable pieces are how much testing we want to do and if we want infrastructure set up, how long it will take to spin it up and push a PR through it (submit queue -jenkins' test suites) This is all potentially variable depending on what the fix itself is, though I suspect we probably don't want to be figuring that out on the fly during the event itself and would rather just have our usual, complete infrastructure ready to go. We can hammer this out in more detail in a PRD but I'd like to get a sense of a realistic target and skeleton plan here. Testing is the big question mark here. Thoughts ?
it looks like doesn't have a reviewer. Do we expect to get it submitted soon? Since it's in a different repository, this issue doesn't seem intrinsically tied to the 1.3 milestone, other than that we wanted visibility around getting a fix for this issue. Does it still belong in the 1.3 milestone?
Correct, it is not directly tied to 1.3. Removed.
I don't understand the difference between these releases and normal patch releases. It sounds like these are for emergency releases but I don't understand why they don't get released like normal x.y.z releases. xref
Standard x.y.z releases contain cumulative patches on that branch. The emergency branches are branched from a specific x.y.z release/tag.
understood, got it, that makes sense. Just a little head spiny trying to read I think we need to create better naming and categories for overall. post.
/sig release
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: // standard kubelet launcher) func (e *E2EServices) Start() error { var err error <del> if !framework.TestContext.NodeConformance { if e.services, err = e.startInternalServices(); err != nil { return fmt.Errorf("failed to start internal services: %v", err) } </del> <ins> if e.services, err = e.startInternalServices(); err != nil { return fmt.Errorf("failed to start internal services: %v", err) } klog.Infof("Node services started.") // running the kubelet depends on whether we are running conformance test-suite if framework.TestContext.NodeConformance { klog.Info("nothing to do in node-e2e-services, running conformance suite") } else { </ins> // Start kubelet e.kubelet, err = e.startKubelet() if err != nil { return fmt.Errorf("failed to start kubelet: %v", err) } <ins> klog.Infof("Kubelet started.") </ins> } return nil }

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: "github.com/GoogleCloudPlatform/kubernetes/pkg/admission" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> apierrors "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/latest" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/meta" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/rest" "github.com/GoogleCloudPlatform/kubernetes/pkg/healthz"

If go-restful detects a particular method is not allowed, it will return this error: That is preventing useful methods that resthandler would generate from making it to the client. Expected response:
triggered with
/cc
/cc
Is the right fix to update the client to generate the message versus changing the server to allow the method and report the error? My bias is to improve the client and not change the server as it makes reading our swagger doc easier to not see a bunch of verb unsupported entries. Sent from my iPhone
My bias is also towards not changing the server to keep swagger spec correct. By changing the client, do you mean changing kubectl? can you please clarify how is the json useful to client? It does not contain any useful extra information. On Feb 11, 2015 7:28 PM, "Derek Carr" wrote:
Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to. Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect.
This was an unintended side effect, but I think the old behavior was an unexpected luxury? I agree from the end-user perspective, the old message was nice, and it's great that the client was politely told that a given verb was not supported on the resource, but I don't agree that a server should appear to accept a method to only report that its not allowed. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:04:41 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to.

Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect. Reply to this email directly or view it on GitHub:
If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error?
I think so, this is only going to get worse when we support specific types of verbs on specific resources. In practice, it seems fine for kubectl to give a better error specific to the method not allowed response. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:26:08 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error? Reply to this email directly or view it on GitHub:
We could consider submitting a PR to go-restful to allow us to provide an error-generation function or interface implementation.
Yeah, that seems reasonable.
Opened
Copying text from : Some errors return as json and some as plain text. This causes overhead on client side when parsing the errors since need to recheck content type for every error in order not to parse plain text as json (some libraries fail on that). The errors message format should be standardized. Examples: The text/plain errors are from go-restful. cc
+1, thanks
The status objects are always of the default API version, however, rather than the version requested, because the request object isn't available to the error handler.
Rather than: we need:
Can we define an error handler on a web service?
At the moment, it's at the container level:
David and I talked about keeping the API lookup mechanism so we could use that in the error handler (apirequestinfo). We still need it for some use cases (proxies) so it could be used here to identify the version and proposed resource if matched.


Question: lateBinding:    test.lateBinding, enableTopology: test.topology, <del> // Not strictly necessary, but without it, the external-attacher takes two minutes to detach // the volume?! Looks like a bug. </del> <ins> // Not strictly necessary, but runs a bit faster this way // and for a while there also was a problem with a two minuted delay // due to a bug (https://github.com/kubernetes-csi/csi-test/pull/250). </ins> disableAttach:  true, registerDriver: true, }

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: CSI mock volume storage capacity exhausted, late binding, with topology Testgrid link: Reason for failure: Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated
/assign
According to , there really are extra calls to and that shouldn't be there: The current theory is that this might be related to kubelet assuming that a CSI driver does real mounts, something that the mock driver (and potentially other) CSI drivers don't do. A fix for that is pending in That this test runs into this randomly is caused by the test doing more strict checking of CSI calls than other tests. The flaky test failures could be avoided by relaxing that checking, but that won't fix the underlying issue.
I'm not sure I would pin it on the mount check yet. The broader question is why is MountVolume() getting called twice after the first one returns success.
is it due that, the timeout of CSI transaction or call is very low where CSI driver takes more time than this timeout?


Question: RegisterFakeList("namespaces", &cluster2Client.Fake, &api_v1.NamespaceList{Items: []api_v1.Namespace{}}) cluster2CreateChan := RegisterFakeCopyOnCreate("namespaces", &cluster2Client.Fake, cluster2Watch) <ins> RegisterFakeList("replicasets", &fakeClient.Fake, &extensionsv1.ReplicaSetList{Items: []extensionsv1.ReplicaSet{ { ObjectMeta: api_v1.ObjectMeta{ Name:      "test-rs", Namespace: ns1.Namespace, }}}}) RegisterFakeList("secrets", &fakeClient.Fake, &api_v1.SecretList{Items: []api_v1.Secret{ { ObjectMeta: api_v1.ObjectMeta{ Name:      "test-secret", Namespace: ns1.Namespace, }}}}) RegisterFakeList("services", &fakeClient.Fake, &api_v1.ServiceList{Items: []api_v1.Service{ { ObjectMeta: api_v1.ObjectMeta{ Name:      "test-service", Namespace: ns1.Namespace, }}}}) nsDeleteChan := RegisterDelete(&fakeClient.Fake, "namespaces") rsDeleteChan := RegisterDeleteCollection(&fakeClient.Fake, "replicasets") serviceDeleteChan := RegisterDeleteCollection(&fakeClient.Fake, "services") secretDeleteChan := RegisterDeleteCollection(&fakeClient.Fake, "secrets") </ins> namespaceController := NewNamespaceController(fakeClient) informer := ToFederatedInformerForTestOnly(namespaceController.namespaceFederatedInformer) informer.SetClientFactory(func(cluster *federation_api.Cluster) (kube_release_1_4.Interface, error) {

In PR the specific resource types are handled explicitly. Need to delete them in more generic way, in a similar way that namespace controller does.
- switching this to 1.6 as it's too late for 1.5. ok? (please switch it right back if you disagree)
did fix this?
Please close or move to next-candidate with work remaining to be done.
Yes fixed by


Question: Filter: &schedulerapi.PluginSet{ Enabled: []schedulerapi.Plugin{ {Name: nodeunschedulable.Name}, <del> {Name: noderesources.FitName}, </del> {Name: nodename.Name}, <del> {Name: nodeports.Name}, </del> <ins> {Name: tainttoleration.Name}, </ins> {Name: nodeaffinity.Name}, <ins> {Name: nodeports.Name}, {Name: noderesources.FitName}, </ins> {Name: volumerestrictions.Name}, <del> {Name: tainttoleration.Name}, </del> {Name: nodevolumelimits.EBSName}, {Name: nodevolumelimits.GCEPDName}, {Name: nodevolumelimits.CSIName},

Inspired from Our default Filter plugins have the following order: IIUC, this order was chosen to have the fastest Filters or the most probability of reporting Unschedulable first. Arguably, some are pretty much equivalent and could be reordered. The major problem with this order is that plugin is almost at the beginning. This plugin only reports . A Node that fails this plugin will be evaluated during preemption. But perhaps this Node doesn't satisfy other UnschedulabeAndUnresolvable requirements (taints, node selectors, nodeaffinity). Theoretically, having those Filters run first could improve preemption latency. The only open question is whether or not the ordering we have today is an API guarantee. I believe it's not (at least in component config), because the only way to change the ordering of Plugins is disabling all of them first. Decided order: /sig scheduling
for comment
and :)
Even we changed the order of plugins, the framework.RunFilterPlugins still exits early when returned by plugin. So should we change the current behavior to run the filter plugins until there is one plugin returned ?
That is correct, and that's what it should be.
No, that's what the original conversation suggested and it's likely not ideal in the average case.
+1 to reordering. Taints, affinity, node name can all be executed before checking for resource fit. No, it is not.
+1. Lightweight plugins that may return status can be moved ahead of other plugins that only return status.
FYI: from who also had this concern: So how about , is it also a good candidate to be moved ahead of ? NodeAffinity and NodeSelector both incorporated in , I'm pretty sure NodeSelector logic is simply, but not 100% sure NodeAffinity logic falls into the category of "lightweight".
Let me propose an order, for extra clarity:
internally composes and invokes , which IMO is a bit more expensive than . should be cheaper than . So my 2 cents is:
Both proposals are fine by me.
/triage accepted /help /good-first-issue
/assign
I want to help here :)
/remove-help


Question: valueFrom: fieldRef: fieldPath: metadata.namespace <del> image: gcr.io/google_containers/cassandra:v4 </del> <ins> image: gcr.io/google_containers/cassandra:v5 </ins> name: cassandra ports: - containerPort: 9042

The namespace seems to be hard coded :
cc:
Thanks. We can improve it after v1 release.


Question: "//cmd/kubeadm/app/util/staticpod:go_default_library", "//pkg/kubeapiserver/authorizer/modes:go_default_library", "//pkg/master/reconcilers:go_default_library", <ins> "//pkg/util/pointer:go_default_library", </ins> "//pkg/util/version:go_default_library", "//vendor/k8s.io/api/core/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/util/sets:go_default_library",

Dear All: I'm not sure if I'm right. When i use whith .I found controller-manager logs Someone can help me? Thanks a lot.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please give more information about your set up cc
Sorry for report later. My openstack didn't support metadata service,but support config-drive service. More info:
Thanks Based on this, I think it's an openstack cloudprovider problem more than a kubeadm problem...
Thank you for help me. I will change the label.
/area cloudprovider
/remove-kind support
Should be fixed with you can specify a search order for metadata and config drive /close
Specifying a search order doesn't really solve the problem, especially if the metadata api isn't available which is what ZhenyangZhao said . I'm getting the same issue and since the config drive is my only option at the moment.
Like for this to be /open
can you please provide logs/versions for your problem in your environment? Thanks, Dims
Let me know if you need anything else Kubernetes version: 1.8.4 Docker version: 17.09.0-ce Linux: CentOS 7.4 Looking at the docker container log for the api-server (I don't have problems with the controller-manager):
and share the following:
Hi! Thank you for filing this issue. We appreciate your interest in Kubernetes and kubeadm. In order to make the kubernetes/kubernetes issue tracker more usable, we're routing issues to different repositories with different focuses. I think this issue belongs in the kubeadm repository. Please move this issue to the in order for the kubeadm team to be able to track this efficiently. If you disagree with my assessment of moving this issue, please speak up and/or reopen the issue. Finally, please try to reproduce the bug with the latest Kubernetes version, it might as well already have been fixed :) Thank you and have a good day!


Question: // the underlying connection (like WebSocket or HTTP2 clients). Pure HTTP clients should use // the higher level TransportFor or RESTClientFor methods. func HTTPWrappersForConfig(config *Config, rt http.RoundTripper) (http.RoundTripper, error) { <del> cfg, err := config.transportConfig() </del> <ins> cfg, err := config.TransportConfig() </ins> if err != nil { return nil, err } return transport.HTTPWrappersForConfig(cfg, rt) } <del> // transportConfig converts a client config to an appropriate transport config. func (c *Config) transportConfig() (*transport.Config, error) { </del> <ins> // TransportConfig converts a client config to an appropriate transport config. func (c *Config) TransportConfig() (*transport.Config, error) { </ins> wt := c.WrapTransport if c.AuthProvider != nil { provider, err := GetAuthProvider(c.Host, c.AuthProvider, c.AuthConfigPersister)

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
We understand the problem here - I'm working on the fix.
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}


Question: "v1.PodAffinityTerm": { "id": "v1.PodAffinityTerm", "description": "Defines a set of pods (namely those matching the labelSelector relative to the given namespace(s)) that this pod should be co-located (affinity) or not co-located (anti-affinity) with, where co-located is defined as running on a node whose value of the label with key u003ctopologyKeyu003e matches that of any node on which a pod of the set of pods is running", <ins> "required": [ "topologyKey" ], </ins> "properties": { "labelSelector": { "$ref": "v1.LabelSelector",

The field is marked as optional: The comment also describes a use case for empty topologykey: But the validation rule says it's required:
cc
cc could you take a look? Thank you. The validation rule is by
the release note of should go into the "action-required" section of the , but it's currently in "Other notable changes".
/assign
It seems that we should also update the external api type comments. Right?
I will look into it today.
If you didn't work on this, I am pleasure to send PR to fix this. :)
yeah sure. thanks.


Question: "fmt" "io" "io/ioutil" <ins> "os" </ins> "strings" "github.com/golang/glog"

Kubernetes version v1.3.0 Kubelet concatenates host and cluster domain entries for my pod, resulting in an invalid config (too many entries with duplicates). The max is only 6 search domains. Any more and DNS resolution breaks when trying to leverage search domains. The example below has only cluster DNS entries, but illustrates the issue: Host search nameserver 10.233.0.2 nameserver 10.90.0.1 options timeout:2 options attempts:2 Container kubectl run -i --tty busybox --image=busybox -- sh (skip waiting text) / # cat search nameserver 10.233.0.2 options attempts:2 options ndots:5 I should be enable cluster DNS config in my host DNS settings and still have functional pod DNS. Kubelet should be intelligent enough to filter out duplicate search domain entries (or drop search domains beyond the 6th entry)
I think some sane checks needs to be too. libc limits search field to 6 records and 255(?) characters. This limitation should be checked before creation.
Hey guys, I'm working on it.
Dup of


Question: if err != nil { return err } <del> return pprof.StartCPUProfile(f) </del> <ins> err = pprof.StartCPUProfile(f) if err != nil { return err } </ins> // Block and mutex profiles need a call to Set{Block,Mutex}ProfileRate to // output anything. We choose to sample all events. case "block": runtime.SetBlockProfileRate(1) <del> return nil </del> case "mutex": runtime.SetMutexProfileFraction(1) <del> return nil </del> default: // Check the profile name is valid. if profile := pprof.Lookup(profileName); profile == nil {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: While investigating I've found that kubectl doesn't handle sigterm with profiling. This way, when profiling a long run command (as wait, drain, etc) and interrupting the command in the middle (ctrl-c, as an example) the profile file is not available. What you expected to happen: Profiling file exists if the command exists correctly or if the command is interrupted with ctrl-c How to reproduce it (as minimally and precisely as possible): Execute a long run command (as kubectl drain, kubectl wait) with --profile=cpu and then interrupt the command before it finishes the execution Anything else we need to know?: Because kubectl deals with errors with an util (cmdutil.CheckErr) that in some specific time exists with os.Exit(1), it's not possible yet to also profile programs that exits. Environment: Kubernetes version (use ): kubectl compiled from upstream Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 20.04
/sig cli /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "k8s.io/kubernetes/pkg/kubelet/token" kubetypes "k8s.io/kubernetes/pkg/kubelet/types" "k8s.io/kubernetes/pkg/kubelet/userns" <ins> "k8s.io/kubernetes/pkg/kubelet/userns/inuserns" </ins> "k8s.io/kubernetes/pkg/kubelet/util" "k8s.io/kubernetes/pkg/kubelet/util/manager" "k8s.io/kubernetes/pkg/kubelet/util/queue"

is no longer compilable on macOS in the master branch. It was compilable with Kubernetes v1.29.0 and v1.30.0-alpha.0. It should be compilable. No response <detailsmaster () (This is just the latest commit, not the commit that introduced the regression) </details<detailsN/A </details<detailsmacOS 14.2.1 (Intel) </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Regression in:


Question: syncPeriod    time.Duration minSyncPeriod time.Duration // Values are CIDR's to exclude when cleaning up IPVS rules. <del> excludeCIDRs   []string </del> <ins> excludeCIDRs []string // Set to true to set sysctls arp_ignore and arp_announce strictARP      bool </ins> iptables       utiliptables.Interface ipvs           utilipvs.Interface ipset          utilipset.Interface

/kind bug Kubernetes version: 1.13.1. kube-proxy mode: IPVS. SDN: flannel. We are used services like this: where IP 192.168.10.201 is from our internal /24 network (in particular, our Kube nodes have IPs 192.168.10.1, 192.168.10.2 etc). Note that there is no conflicts between real IPs and virtual external ones. Before updating from 1.12.3 to 1.13.1, it worked correctly. In partucular these IPs was accessible from other non-Kubernetes hosts from 192.168.10.0/24 network. After update kube-proxy to 1.13.1, this feature stops working: non-Kubernetes hosts cannot resolve service IPs via ARP: On Kubernetes hosts it still working, because not requires ARP resolving. If we reset sysctl parameter arpignore on Kube hosts, all works normally even from non-Kube hosts. Quck debugging shows that problem is in the commit . Please make this behaviour opt-out. (Now we resets arpignore to 0 via cron every 1 min.)
/sig network
Having the same problem on Kubernetes 1.13.2 btw Thanks for a workaround!
Same problem, fix this please!!!
This is similar to We've had a discussion there on how to best implement a solution but haven't decided yet I'm in favor of flag which defaults to the previous behavior to avoid breaking existing setups (alternatives are cleaner but involve creating an additional arptable rule per service which I think we should avoid) what do you think?
OK

What happened: After PR: , we noticed communications issue in our pods. We use the Lyft CNI plugin () which relies on two interfaces in pod: an IPVLAN interface connecting to the AWS VPC which has a VPC IP an Unnumbered point-to-point interface (veth) connecting to the host and that does not have an IP (it "borrows" the one from the IPVLAN interface) On the host side, there is a veth interface which also doesn't have an IP (it borrows one from the main interface). With the latest versions of kube-proxy communication between host and pods break What you expected to happen: Communication still works How to reproduce it (as minimally and precisely as possible): Use the Lyft CNI plugin Deploy kube-proxy Try to ping the host from a pod (or a pod from the host) Anything else we need to know?: The new sysctls create 2 issues: containers started after kube-proxy inherit so they will not answer ARP queries on the veth interface the veth on the host side also does not have an IP so it will no answer ARP queries from the pods I know that our setup is pretty uncommon but assuming that the veth on the host side will have an IP seems dangerous. Some discussions on the PTP plugin for instance suggest not using IP for the host interface: containernetworking/plugins Possible fixes: create a flag to enable ARP filtering (I think to avoid breaking existing setups, we should disable it by default but I'm fine with enabling it by default as long as we document the solution) use arptables to filter ARP queries for clusterIPs (): a bit more work because we need to do it for all clusterIPs (there may be a more efficient way to do it but I could not make it work by filtering on the kube-ipvs interface) and this requires the arptables binary. We could test for the existence of the binary and log a warning if it is missing (similar to behavior with conntrack) Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign ping
I can say only that solution with arptables working fine with MetalLB.
Oh, really? Glad you confirm that.

But I would like to make sure that we won't create too many arptables rules. Can ipset works with arptables?
It doesn't look like arptables works with ipset
Unfortunately doesn't support ipsets, but new do. (not fully implemented yet) Eg we can create ipset for arp: But current implementation is not allowing to use it for arp rules for now: Will return: I guess because there is still no type specified for (destination ip) and (source ip) BTW: nft does not allow to create set of integer components: But there is a way for specify multiple IPs in one rule, eg: Where is IPv4 addresses in decimal format More info here:
Another option would be to use the and filter all queries but this flag is not in kube-proxy today
Bad idea, it will handle but not - which can be really differ
can you explain more detail why Lyft CNI can not assign IPs to the right interfaces for avoid this kind of issues?
Yes you are right. bad idea It could definitely add IPs to these interfaces but it's not the choice that was made. More details here: ("Unnumbered point-to-point interface"). I could probably PR a fix there. I'm a bit worried that some other CNI plugins could break here though
We are about to deploy some 1.13 clusters and can't use kube-proxy 1.13 as is We could start with a custom build but I assume others will be impacted The simplest way forward for now is probably to use a flag to control this behavior. I think the safest is probably to disable this by default to avoid the issue for people upgrading but if you think it is really important we can enable it by default. I know we'd like to avoid adding more flags but other solutions seem a bit more complicated and will most likely result in one iptables rule per service. What do you think ?
Just FYI, we have another CNI plugin that is affected by this.
We addressed the issue by assigning a link-local address 169.254.1.1/32 to the host-side veth in . So this issue is not critical at least for our case.
Good news everyone, new nft supports setting ip address for arp filters : that's makes possible for using sets


Question: }, }, }, <ins> { accept: runtime.ContentTypeJSON + ";as=Table;v=v1alpha1;g=meta.k8s.io", params: url.Values{"includeObject": []string{"Metadata"}}, expected: &metav1alpha1.Table{ TypeMeta: metav1.TypeMeta{Kind: "Table", APIVersion: "meta.k8s.io/v1alpha1"}, ColumnDefinitions: []metav1alpha1.TableColumnDefinition{ {Name: "Name", Type: "string", Description: metaDoc["name"]}, {Name: "Created At", Type: "date", Description: metaDoc["creationTimestamp"]}, }, Rows: []metav1alpha1.TableRow{ {Cells: []interface{}{"foo1", now.Time.UTC().Format(time.RFC3339)}, Object: runtime.RawExtension{Raw: encodedBody}}, }, }, }, </ins> } for i, test := range tests { u, err := url.Parse(server.URL + "/" + prefix + "/" + testGroupVersion.Group + "/" + testGroupVersion.Version + "/namespaces/default/simple/id")

Making an API request for table output and specifying the parameter causes a server error in parameter conversion, and an inaccurate error message in response: server log: E0804 09:41:14. ] apiserver received an error that is not an metav1.Status: converting (url.Values).[]string to (v1alpha1.TableOptions).v1alpha1.IncludeObjectPolicy: couldn't copy '[]string' into 'v1alpha1.IncludeObjectPolicy'; didn't understand types
Ugh, typed strings.


Question: "github.com/Azure/azure-sdk-for-go/services/network/mgmt/2017-09-01/network" "github.com/Azure/go-autorest/autorest" "github.com/golang/glog" "k8s.io/apimachinery/pkg/types" cloudprovider "k8s.io/cloud-provider" )

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When creating a Kubernetes 1.12.1 cluster in Azure with Availability Set backed masters and VMSS backed agents, the cloud-controller-manager is unable to parse the ProviderID returned by the Azure SDK. What you expected to happen: The providerID should be split and the cloud-controller-manager should be able to find a kubelet node IP from the cloud provider How to reproduce it (as minimally and precisely as possible): Stand up a master node in Azure within an availability set with the kubelet configured with Run the configured for Azure Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig azure
When switching over to vmType the following was encountered;
The fix in resolved the issue that was seen in this issue, closing


Question: # Automatically detect swift url if it wasn't specified if [[ -z $SWIFT_SERVER_URL ]]; then <del> SWIFT_SERVER_URL=$(openstack catalog show object-store --format value | egrep -o "publicURL: (.+)$" | cut -d" " -f2) </del> <ins> local rgx="" if [ "$OS_IDENTITY_API_VERSION" = "3" ]; then rgx="public: (.+)$" else rgx="publicURL: (.+)$" fi SWIFT_SERVER_URL=$(openstack catalog show object-store --format value | egrep -o "$rgx" | cut -d" " -f2) </ins> fi local swift_repo_url="${SWIFT_SERVER_URL}/kubernetes"

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Internal OpenStack cloud OS (e.g. from /etc/os-release): OS X 10.10.5 Kernel (e.g. ): Darwin Sean-Collins- 14.5.0 Darwin Kernel Version 14.5.0: Sun Sep 25 22:07:15 PDT 2016; root:xnu-2782.50.9~1/RELEASEX8664 x86_64 Install tools: Others: What happened: kube- failed to create the heat stack KubernetesStack, and exited What you expected to happen: Create the stack KubernetesStack and continue onward How to reproduce it (as minimally and precisely as possible): See above console output Anything else do we need to know:
My suspicion is that the behavior of and the command has changed recently when
Nope, it actually looks like it's the (kubernetes)scollins ~/src/k8s/kubernetes ±swiftuploadcache » openstack catalog show object-store +-----------+
--------+ +-----------+
--------+ +-----------+
--------+ (kubernetes)scollins ~/src/k8s/kubernetes ±swiftuploadcache » openstack catalog show object-store --format value cut -d" " -f2 (kubernetes)scollins ~/src/k8s/kubernetes ±swiftuploadcache »
I'll need to check to see if the change from to is a server side change from OpenStack Keystone, or a change in , or is something that is deployed incorrectly in this environment
So, at least according to the , endpoints have an - which can be , , or - never
Yeah, so is from the


<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic EBS volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. With scale, this can trigger massive from ec2, and affect cluster stability. This is also related to What you expected to happen: To simply get zones with available instances, we can get information from API server for efficiency. There might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe. How to reproduce it (as minimally and precisely as possible): We only expect this to be hit when the cluster is huge (or aws account is huge) so not easy to reproduce. /assign /sig aws /sig cloud-provider
/assign


Question: import ( "context" "path/filepath" <ins> "regexp" </ins> "time" <ins> "github.com/onsi/ginkgo" "github.com/onsi/gomega" </ins> appsv1 "k8s.io/api/apps/v1" v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/serializer" <del> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" </del> <ins> kubeletdevicepluginv1beta1 "k8s.io/kubelet/pkg/apis/deviceplugin/v1beta1" </ins> e2etestfiles "k8s.io/kubernetes/test/e2e/framework/testfiles" admissionapi "k8s.io/pod-security-admission/api" <del> "regexp" </del> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid"

ci-kubernetes-node-kubelet-serial-containerd: ci-kubernetes-node-kubelet-serial: (This is just an example, most of the tests appear to be failing) /sig node /kind failing-test /milestone v1.23 since test is failing in a release-blocking job /priority critical-urgent /assign /cc I believe this stems from
/triage accepted These failures look weird - I don't understand why dockershim is sometimes failing after restarting the kubelet. I guess we should double check that timings match the old GPU test (I think I might've tightened them a bit, bc they were passing when I ran the device suite on the same nodes). What's weirder is that they passed on my PR and all of a sudden broke after merging
Discussed at today - the easiest solution now that 1.23 has been branched is to punt these tests out of the serial lane so we can get it green again for 1.23, since any changes at this point must be backported and we would prefer that this is not release-blocking.
/milestone clear should address this; we won't get a fresh serial run for another ~6 hours so let's confirm next business day that this is temporarily fixed.
/remove-priority critical-urgent /priority important-longterm
can we keep it in the milestone until we get a verified green run?
CI is green today after skipping DevicePluginProbe tests .
/assign Maybe related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
Looks like serial has been stable for a while, closing: /close
Closing this issue. <detailsIn response to : Looks like serial has been stable for a while, closing: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen it is referred to here: Tests are skipped today
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
did you have a chance to look at this? I remember you were close to understanding on what is going wrong here
oh welp, missed that
I was (past tense) but now I need to go back and look at my notes. If someone else wants to take over, be my guest, I've been quite busy with other tasks lately. Otherwise I'll get back to this!
So! I'm able to resume working in this area. So I'm testing last k/k master branch tip and running with : and the tests seems to be running and passing quite reliably! (5 times in a row seems like a promising start :) ) To move forward I suggest to: a explicit s in . This will make more evident we have a pending issue in our camp, that's the most egregious exception we have and it's confusing the skip in the config: - this will enable us to do focused work in this area a PR to remove the skip and iterate over it running CI until it's good enough I can take care of all the above. If CI is still failing, we need to explore possible missing/wrong configuration, but then other e2enode tests should be failing, not just this one.
I agree, this plan sounds good. Thank you!

xref:
started to fail also locally - we have baremetal lanes in RH - in the project CI they skip the relevant test. Investigating a possible common root cause
after a very long troubleshooting and quite few attempts, is ready for review


Question: oidctesting "k8s.io/kubernetes/plugin/pkg/auth/authenticator/token/oidc/testing" ) <del> func generateToken(t *testing.T, op *oidctesting.OIDCProvider, iss, sub, aud string, usernameClaim, value, groupsClaim string, groups []string, iat, exp time.Time) string { </del> <ins> func generateToken(t *testing.T, op *oidctesting.OIDCProvider, iss, sub, aud string, usernameClaim, value, groupsClaim string, groups interface{}, iat, exp time.Time) string { </ins> signer := op.PrivKey.Signer() claims := oidc.NewClaims(iss, sub, aud, iat, exp) claims.Add(usernameClaim, value)

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No, already discussed on sig-auth What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): confirmed issue on sig-auth channel Is this a BUG REPORT or FEATURE REQUEST? (choose one): Depends on interpretation <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-07-01T19:26:38Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.4+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-08-02T00:54:53Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: CoreOS Vagrant Image OS (e.g. from /etc/os-release): NAME=CoreOS ID=coreos VERSION=1122.0.0 VERSIONID=1122.0.0 BUILDID=2016-07-27-0739 PRETTYNAME="CoreOS 1122.0.0 (MoreOS)" ANSICOLOR="1;32" HOMEURL="" BUGREPORTURL="" Kernel (e.g.

): Linux localhost 4.7.0-coreos SMP Wed Jul 27 07:30:04 UTC 2016 x8664 Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz GenuineIntel GNU/Linux Install tools: Vagrant Others: What happened: The claim named in --oidc-groups-claim MUST be list of strings, even if there is only one value. Talking to this is by design since its assumed that users will have more then one group. What you expected to happen: The OIDC authenticator should accept either a list of strings or a single string since: There is no concept of schema that marks an attribute as single value or multi value in JSON or a JWT There is no standard for representing groups in a JWT There is no standard for representing groups in OIDC An OIDC should be flexible in how it accepts data. How to reproduce it (as minimally and precisely as possible): Deploy with RBAC enabled Create a role binding for the group "admin" Generate a JWT such as the one that does NOT work below User won't be able to login Generate a JWT where the role is multi-value User will be authorized Does NOT Work: DOES Work: Anything else do we need to know:
These should be called groups, not roles, right? Those are different. And, is it normal for a JWT to contain keys which are arrays? That seems weird.
where do you see role defined? you mean the user_role claim?
Claims can be structured, yes. The name of the groups claim is arbitrary. Could easily have been "group". In my mind, this is all about enabling mapping a "groupish" claim from whatever IDP issued the JWT, even if it is a single value instead of an array.
That's the way it works currently. Yeah I'm fine with this.
Okay.


Question: } func (f *fakePetClient) setHealthy(index int) error { <del> if len(f.pets) < index { </del> <ins> if len(f.pets) <= index { </ins> return fmt.Errorf("Index out of range, len %v index %v", len(f.pets), index) } f.pets[index].pod.Status.Phase = api.PodRunning

Is this a request for help? Nope What keywords did you search in Kubernetes issues before filing this one? petset replicas current desired Is this a BUG REPORT or FEATURE REQUEST? BUG Kubernetes version Client Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2016-09-01T10:49:36Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"", Minor:"", GitVersion:"v0.0.0-master+", GitCommit:"", GitTreeState:"not a git tree", BuildDate:"2016-09-01T10:49:36Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: 24 cores, 64 GB RAM OS Arch Linux Kernel (e.g. ): Linux hostname 4.4.13-1-lts SMP Wed Jun 8 16:44:31 CEST 2016 x86_64 GNU/Linux Install tools: local-cluster- Others: What happened: While creating PetSet based on this tutorial: first PetSet pod was stuck in pending state due to lack of PersistentVolumes in cluster. When getting the petset info via I expected to see "CURRENT" set to 1, while the kubectl output was as follows: While clearly shows that only one pod is created: In json and yaml output, status.replicas value was also 2. What you expected to happen: I expected that CURRENT value shown by kubectl would be due to only one pod being created. How to reproduce it (as minimally and precisely as possible): Create via on empty cluster and run and . Anything else do we need to know: Nope
CURRENT is the number of pods that have been created, not that they are READY. See for example the latest api change for replica sets where we exactly for this reason. All controllers (petsets, daemonsets, deployments, etc.) should have ReadyReplicas cc:
See also
I guess it doesn't work anyway, since only one pod was created for this PetSet, but the CURRENT says that 2 were created. I will change bug description then.

Hmm, the idea is to sync every pet periodically, but only create them one at a time. The logic that counts status.replicas is wrong. This counter shouldn't be incremented in the case where we didn't update/sync or create a pet: Today we do increment it all pet creation is blocked on the first pet becoming unhealthy. Should be an easy fix if anyone wants to jump in ( fyi) otherwise I'll get to it once 1.4 is out.
I can try to fix it if that's OK. Thanks for the pointer to the source code!
Please do, co-assigned


Question: }, }, Spec: api.PodSpec{ <ins> Affinity: loaderAffinity, </ins> Containers: []api.Container{{ Name:    "test", Image:   busyboxImage,

The "AppArmor load AppArmor profiles should enforce an AppArmor profile" e2e test in failing on our 5k-node gce clusters - https://k8s- (run no. 8) due to timeout: Seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. It's passing on our 2k-node clusters (https://k8s-) but failing for 5k - suggesting that it's likely due to size of the cluster. I think it's the ds creation taking longer on large cluster. cc
Seems like you're the most relevant here. Could one of you take a look into it? And does increasing the timeout for the test-pod based on no. of nodes sound reasonable?
Assigning it you since is OOO. Feel free to reassign as apt.
This is part of the correctness suite and large cluster tests are release-blocking.
FYI
Reassigning to who is back on Tuesday. He wrote the AppArmor stuff and should be able to debug this quickly. /assign
Ref
Could you manage to take a look into this?
Sorry, I meant to reply to this! I have a fix, just need to confirm it works. I think you're right about the DaemonSet timing out. There's no reason this needs to load the profiles on every node, so the fix just runs a single loader, and uses pod affinity to co-schedule the test pod.
Smooth!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: func (proxier *metaProxier) OnEndpointsDelete(endpoints *v1.Endpoints) { ipFamily, err := endpointsIPFamily(endpoints) if err != nil { <del> klog.Warningf("failed to delete endpoints %s/%s with error %v", endpoints.ObjectMeta.Namespace, endpoints.ObjectMeta.Name, err) </del> <ins> klog.V(4).Infof("failed to delete endpoints %s/%s with error %v", endpoints.ObjectMeta.Namespace, endpoints.ObjectMeta.Name, err) </ins> return }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Setup a kubernetes cluster with dual stack enabled, set kube-proxy to "ipvs" proxy mode, many warnings are flooding in kube-proxy pod log, W0303 22:11:22. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:22. 1 ] failed to update endpoints kube-system/kube-controller-manager with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:24. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:24. 1 ] failed to update endpoints kube-system/kube-controller-manager with error failed to identify ipfamily for endpoints (no subsets) W0303 22:11:26. 1 ] failed to update endpoints kube-system/kube-scheduler with error failed to identify ipfamily for endpoints (no subsets) What you expected to happen: No such error logs How to reproduce it (as minimally and precisely as possible): Here is my setup step, I'm using a single master node setup, kubeadm init --pod-network-cidr=192.168.0.0/16,fc00::/112 --service-cidr=10.96.0.0/16,fd00::/112 --feature-gates=IPv6DualStack=true Change controller-manager cidr configuration --node-cidr-mask-size-ipv6=120 install calico 3.6.4 with dual stack enabled kube-proxy proxy mode set to "ipvs" featureGates: IPv6DualStack: true healthzBindAddress: "" hostnameOverride: "" iptables: masqueradeAll: false masqueradeBit: null minSyncPeriod: 0s syncPeriod: 0s ipvs: excludeCIDRs: null minSyncPeriod: 0s scheduler: "" strictARP: false syncPeriod: 0s kind: KubeProxyConfiguration metricsBindAddress: "" mode: "ipvs" Anything else we need to know?

: Linux Redhat 7.7 Environment: Kubernetes version (use ):v1.17.3 Cloud provider or hardware configuration: OS (e.g: ): Redhat 7.7 Kernel (e.g. ): 3.10.0-1062.4.1.el7.x8664 Install tools: kubeadm Network plugin and version (if this is a network-related bug): calico 3.6.4 Others:
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/assign
/remove-triage unresolved those warnings are harmless because those objects doesn't have endpoints addresses, thus no IP family, as you can see below: however, I think that if there are no endpoint addresses we should skip instead of logging, what do you think ?
It seems these printouts can be caused by headless services, please see;
Can endpoints support dual stack? From my understanding, it cannot support dual stack in v1.17. Why kube-proxy will check another ipFamily which is not configured? Here is the endpoints info under default ns from my cluster,
Another observation about dual stack supporting is I don't see ipv6 interface address in node status. Here is my interface configuration, and here is pod configuration, I see kubelet has already set ipv6 network in podCIDRs
The is reported by if you specify . Currently kubelet does not report both ipv4 and ipv6 addresses, that is not implemented yet. But public clouds do not let report the node but defines them themselves. At least set both addresses. So it depends on your environment.
endpoints on services with ClusterIP services are single family, with the same IPFamily than the ClusterIP. However, you can create endpoints manually, not associated to a service, or to a headless service (ClusterIP=None) with a mix of families. You opened the issue asking why do you have these errors logs If you check the endpoints of the services mentioned in the logs there are no addresses, hence the kube-proxy logs a WARNING because is not able to identify the IP family. Technically the warning is correct, but I think that if there are no addresses we shouldn't log the warning.

In order to not convert this issue in a placeholder for dual stack let's try to stick to the topic, for more general things we have a #k8s-dual-stack channel in slack.
Ok, I understand what you mentioned. From the implementation, this log behavior is correct. The other problem is kube-proxy is constantly logging the same error.
I think (please somebody correct me if I'm wrong) this is because the endpoints are checked periodically, per the control-loop / reconcile paradigm that kubernetes uses, so it tries to add those endpoints periodically
nodeIPs are only dual stack if the cloud-controller-manager updates it, on bare metal or without cloud-controller-manager you only see one address. I tried cloud-controller-manager in my openstack cluster and can get ipv4/ipv6 interface.
Thanks, I will create another issue for document.


Question: ctx, cancel := watchtools.ContextWithOptionalTimeout(ctx, 0*time.Second) defer cancel() <del> preconditionFunc := func(store cache.Store) (bool, error) { _, exists, err := store.Get(&metav1.ObjectMeta{Namespace: ns, Name: podName}) if err != nil { return true, err } if !exists { // We need to make sure we see the object in the cache before we start waiting for events // or we would be waiting for the timeout if such object didn't exist. // (e.g. it was deleted before we started informers so they wouldn't even see the delete event) return true, errors.NewNotFound(corev1.Resource("pods"), podName) } return false, nil } </del> fieldSelector := fields.OneTermEqualSelector("metadata.name", podName).String() lw := &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: * [component/CLI](https://github.com/GoogleCloudPlatform/kubernetes/labels/component%2FCLI) * [component/client](https://github.com/GoogleCloudPlatform/kubernetes/labels/component%2Fclient) <del> 1. Kind-based filtering on object streams -- only operate on the kinds of objects specified. This would make directory-based kubectl operations much more useful. Users should be able to instantiate the example applications using `kubectl create -f <example-dir> ...` </del> 1. Create services before other objects, or at least before objects that depend upon them. Namespace-relative DNS mitigates this some, but most users are still using service environment variables. [#1768](https://github.com/GoogleCloudPlatform/kubernetes/issues/1768) <del> 1. Pretty printing of endpoints 1. Service address/port lookup command(s) </del> 1. Finish rolling update [#1353](https://github.com/GoogleCloudPlatform/kubernetes/issues/1353) 1. Friendly to auto-scaling [#2863](https://github.com/GoogleCloudPlatform/kubernetes/pull/2863#issuecomment-69701562) 1. Rollback (make rolling-update reversible, and complete an in-progress rolling update by taking 2 replication controller names rather than always taking a file)

Simple version is done, more complex version needs work. Done, now that readiness corresponds to Running and is tracked by Kubelet status if the readiness hook is set? Done Done Do we want to do a more generic one or a less generic one? OpenShift was looking more at specific topical plugin commands ("add-volume", "set-default-resources"), but not sure how much hand-holding users need. Getting close. Exec has landed and is a replacement for ssh Attach on start remains hard. Implemented
I/we should also file issues for outstanding items without issues, and add lines for newly filed issues (e.g., command flag for run-container). Readiness is not yet used by rollingupdate. With respect to patch on the command line: I was thinking of a generic mechanism, but I could see the case for domain-aware commands, also. I think we need to better factor the command code and library code, and generic functionality vs. domain-aware functionality. Neither should ever convert to the internal representation. Domain-aware functionality (e.g., describe) should target a specific API version. Configuration generators, such as run-container, should be provided as services, or at least as separate plugin commands.
ref
ref
ref
ref
ref
ref