Question: func (mounter *NsenterMounter) GetFileType(pathname string) (FileType, error) { var pathType FileType <del> outputBytes, err := mounter.ne.Exec("stat", []string{"-L", `--printf "%F"`, pathname}).CombinedOutput() </del> <ins> outputBytes, err := mounter.ne.Exec("stat", []string{"-L", "--printf=%F", pathname}).CombinedOutput() </ins> if err != nil { <ins> if strings.Contains(string(outputBytes), "No such file") { err = fmt.Errorf("%s does not exist", pathname) } else { err = fmt.Errorf("stat %s error: %v", pathname, string(outputBytes)) } </ins> return pathType, err }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: When a pod with disk mount PVC is already moved from a node, the PVC is always be in node.volumesInUse, makes StatefulSet pod stuck in ContainerCreating forever. Although I found this issue on Azure, I think it's a common issue. What you expected to happen: How to reproduce it (as minimally and precisely as possible): You could find detailed steps up a k8s cluster with 2 agent nodes(node, node) cordon node up 3 statefulsets (1 replica) with azure disk mount each, and all pods will be scheduled to node uncordon node drain node, and then all 3 pods will be scheduled to node some condition, these all 3 pods will be stuck in ContainerCreating forever I have check status of node, node, the is incorrect in node since all pods have been moved to node, in node should be empty, while all 3 volumes are still in node. Actually there are also ghost volumes in node after I Anything else we need to know?: restart kubelet manually in both node, node would work Environment: Kubernetes version (use ): v1.9.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig node
cc
Hi in v1.9.6, we found volume in is not removed even after pod with that volume has already been moved from the node for a very long time. Have you hit that kind of issue before? Thanks.
Finally I have verified this is cuased by issue in nsenter_mount.GetFileType func, with above PR, the node.volumesInUse list will be correct below are the test result with & without this fix Before fix(it costs 9 min for rescheduling 3 pods with disk mount from one node to another) After fix: (it costs 3 min for rescheduling 3 pods with disk mount from one node to another)


Question: return printer.PrintObj(obj, out) } <ins> if o.list && len(o.outputFormat) > 0 { return fmt.Errorf("--list and --output may not be specified together") } </ins> o.namespace, o.enforceNamespace, err = f.ToRawKubeConfigLoader().Namespace() if err != nil { return err

<!-- Please only use this template for submitting enhancement requests --What would you like to be : add --list option to kubectl annotate command Why is this needed: like kubectl label commandshow annotations quickly
/sig cli These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/assign


Question: } func (r *TokenREST) Create(ctx context.Context, name string, obj runtime.Object, createValidation rest.ValidateObjectFunc, options *metav1.CreateOptions) (runtime.Object, error) { <del> if err := createValidation(obj); err != nil { return nil, err </del> <ins> if createValidation != nil { if err := createValidation(obj.DeepCopyObject()); err != nil { return nil, err } </ins> } out := obj.(*authenticationapi.TokenRequest)

What happened: I'm unable to configure the api-server through a to pass admission requests to my for the resource. I'm adding the following rule to the When performing rollbacks, I'm not seeing any calls to the admission webhook. What you expected to happen: To receive validation requests when performing rollbacks. How to reproduce it (as minimally and precisely as possible): Create a ValidatingAdmissionWebhook and perform a rollback request. Anything else we need to know?: Mentioned this on the #sig-api-machinery channel on Slack, and suggested the error was that the -object never is used: Environment: Kubernetes version (use ): 12.3 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Container Linux by CoreOS 1911.5.0 (Rhyolite) 4.14.84-coreos docker://18.6.1 Kernel (e.g. ): Install tools: Others: /kind bug
/sig apps
/sig api-machinery
Good point, should call .


Question: type ReplicationController struct { TypeMeta     `json:",inline"` DesiredState ReplicationControllerState `json:"desiredState,omitempty" description:"specification of the desired state of the replication controller"` <del> CurrentState ReplicationControllerState `json:"currentState,omitempty" description:"current state of the replication controller"` </del> <ins> CurrentState ReplicationControllerState `json:"currentState,omitempty" description:"current state of the replication controller; populated by the system, read-only"` </ins> Labels       map[string]string          `json:"labels,omitempty" description:"map of string keys and values that can be used to organize and categorize replication controllers"` }

Right now we have haphazard comments regarding which fields are read-only/output-only, and many, such as aren't documented at all. Swagger 2.0 actually supports a property tag, but we're using Swagger 1.2, which does not, officially. The official meaning doesn't match what we want, though. fields must not be included in a PUT, which would be painful for clients. We could simply document our convention, however. We should tag fields as readonly. Maybe we could have go-restful automatically add a comment to the description in that case.
/cc
Are there any known obstacles with upgrading to 2.0?
go-restful (the framework that we are using to generate swagger-spec) does not support 2.0 yet:
Disclaimer: for the doc-fix it day, I will be adding the read-only information into the description. This partially resolves this issue, but not entirely. I will unassign myself after description is done.
Note: users can set the node status when creating a node. This is a bug and should be fixed. After fixing that bug, the corresponding description should be updated.


Question: } } <ins> // Test_updateNodeStatusUpdateNeeded expects statusUpdateNeeded to be properly updated if // updateNodeStatusUpdateNeeded is called on a node that exists in the actual state of the world func Test_updateNodeStatusUpdateNeeded(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } nodeName := types.NodeName("node-1") nodeToUpdate := nodeToUpdateStatusFor{ nodeName:                  nodeName, statusUpdateNeeded:        true, volumesToReportAsAttached: make(map[v1.UniqueVolumeName]v1.UniqueVolumeName), } asw.nodesToUpdateStatusFor[nodeName] = nodeToUpdate // Act err := asw.updateNodeStatusUpdateNeeded(nodeName, false) // Assert if err != nil { t.Fatalf("updateNodeStatusUpdateNeeded should not return error, but got: %v", err) } nodesToUpdateStatusFor := asw.GetNodesToUpdateStatusFor() if nodesToUpdateStatusFor[nodeName].statusUpdateNeeded { t.Fatalf("nodesToUpdateStatusFor should be updated to: false, but got: true") } } // Test_updateNodeStatusUpdateNeededError expects statusUpdateNeeded to report error if // updateNodeStatusUpdateNeeded is called on a node that does not exist in the actual state of the world func Test_updateNodeStatusUpdateNeededError(t *testing.T) { // Arrange volumePluginMgr, _ := volumetesting.GetTestVolumePluginMgr(t) asw := &actualStateOfWorld{ attachedVolumes:        make(map[v1.UniqueVolumeName]attachedVolume), nodesToUpdateStatusFor: make(map[types.NodeName]nodeToUpdateStatusFor), volumePluginMgr:        volumePluginMgr, } nodeName := types.NodeName("node-1") // Act err := asw.updateNodeStatusUpdateNeeded(nodeName, false) // Assert if err == nil { t.Fatalf("updateNodeStatusUpdateNeeded should return error, but got nothing") } } </ins> func verifyAttachedVolume( t *testing.T, attachedVolumes []AttachedVolume,

was the fix in sufficient, or do you want to bubble the error past SetNodeStatusUpdateNeeded?
do you want this in v1.7?
can you retitle this to be meaningful or close it?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
I checked the sources, now returns as requested here. /close


Question: return ctlrTestQueueSetCompleter{cqs.cts, cqs, qc}, nil } <ins> func (cqs *ctlrTestQueueSet) UpdateObservations() { } </ins> func (cqs *ctlrTestQueueSet) Dump(bool) debug.QueueSetDump { return debug.QueueSetDump{} }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Not observed yet, but deduced from analyzing the code. If a k/apiserver/pkg/util/flowcontrol/fairqueuing.QueueSet, whose metrics.TimedObserverPair is a pair of k/apiserver/pkg/util/flowcontrol/metrics.sampleAndWaterMarkHistograms, gets zero activity for a long while and then gets some activity, then the loop at will go through a lot of iterations, and do so with a lock held. What you expected to happen: No big hiccups in any scenario. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing I noticed an incorrect fake implementation of Interface. I looked at the comment on the Handle method, and noticed that it does not say when/how the is called. What you expected to happen: I expected the parameter to be as well documented as the others. How to reproduce it (as minimally and precisely as possible): Look at and see how it does not say when/whether/with-what-parameter-values is called. Anything else we need to know?: Environment: Kubernetes version (use ): current Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Any chance we could write a test for the interface which verifies correct plumbing?
/cc
: that should be do-able


Question: expectedAPIConfig: func() *serverstore.ResourceConfig { config := newFakeAPIResourceConfigSource() config.DisableVersions(extensionsapiv1beta1.SchemeGroupVersion) <del> config.DisableResources(extensionsapiv1beta1.SchemeGroupVersion.WithResource("ingresses")) </del> return config }, err: false,

https://k8s- sample failure: (also affecting other CI jobs) cc /sig gcp
/assign We're interested in disabling the job as blocking entirely. We're of the opinion this shouldn't have bee allowed to block the community in the first place. I swear there's a historical issue around tightening the regex somewhere.
... wrong button sorry about that
possibly caused by erroring on invalid values that were previously silently skipped opened to downgrade that error to a warning for a release
Well, I mentioned the idea of making the job optional in
PR out to drop the runifchanged regex from pull-kubernetes-e2e-gke
Confirmed turns the test green
Can call this closed when merges
/pony hooray
Couldn't find a pony matching that query. <detailsIn response to : /pony hooray Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.DeleteAllEtcdKeys() defer framework.DeleteAllEtcdKeys() _, s := framework.RunAMaster(t) <del> defer s.Close() </del> <ins> // TODO: Uncomment when fix #19254 // defer s.Close() </ins> ns := api.NamespaceDefault client := client.NewOrDie(&client.Config{Host: s.URL, ContentConfig: client.ContentConfig{GroupVersion: testapi.Default.GroupVersion()}})

http://pr- is another failure.
Also seen in http://pr-
Again. Not assigned to anyone, so I am sending to for routing
Again: Is there anyone working on this? I will spend some time looking into this.
I think that the name of this issue is misleading - the issue seems to be unrelated to etcd_watcher Basically, if you look into this part of log: So it seems that, the WaitGroup that is problematic is in httptest.Server. So my feeling is that we are simply talking to a server that is already being close and this is ~roughly duplicate of We should just find where exactly the problem is and try to work-around it.
I think this is exactly what we had before. I will send a PR with the fix today.


Question: //  only if the existing object already contains a zero value. //  1:30: lift this restriction, allow zero value via v1 or v1beta3 opts := validation.PriorityLevelValidationOptions{ <del> AllowZeroLimitedNominalConcurrencyShares: utilfeature.DefaultFeatureGate.Enabled(features.ZeroLimitedNominalConcurrencyShares) || hasZeroLimitedNominalConcurrencyShares(oldPL), </del> <ins> AllowZeroLimitedNominalConcurrencyShares: true, </ins> } return validation.ValidatePriorityLevelConfiguration(newPL, getRequestGroupVersion(ctx), opts) }

In 1.30, Priority and Fairness should allow a zero value, either via v1 or v1beta3 (with the roundtrip annotation) in create and update requests See:
/sig api-machinery /cc /assign
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /triage accpted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: plugins[i] = p } <del> w := newWatcherWithHandler(t, hdlr, false /* testDeprecatedDir */) defer func() { require.NoError(t, w.Stop()) }() </del> var wg sync.WaitGroup for i := 0; i < len(plugins); i++ { wg.Add(1)

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: __kubeadm_filedir() { local RET OLD_IFS w qw <del> __debug "_filedir $@ cur=$cur" </del> <ins> __kubectl_debug "_filedir $@ cur=$cur" </ins> if [[ "$1" = ~* ]]; then # somehow does not work. Maybe, zsh does not call this at all eval echo "$1"

/kind bug I wasn't able to find issue opened for ZSH case, and aware about similar, fixed, case with bash in 1.10.0-b1 bash completion is working fine
/sig cli
can you verify this and label for v1.10 milestone and note if it will be considered a release blocker
I cannot reproduce this on my local machine (zsh 5.3.1 x86_64-apple-darwin16.3.0). Please produce a docker container (with Dockerfile) that shows this issue.
And another important question for the release: can you try with the 1.9 kubectl? Is that working in your setup?
little bit more info: i'm running this on CentOS 7.4 1708 x64 and zsh from official repo same issue happenes with 1.10.0-b3: 3
From sig-cli Slack: Kazuki Suda [01:22] It seems that have to be renamed to due to changes contained (edited)
You can reproduce this with the below:
was renamed to at , but has been used yet in completions generated by . So we have to replace to .
issue still exists with v10.0.0-b4
10.0-beta.4 does not contain the fix, yet.
Because the fix is not cherry-picked to release-1.10 yet.


Question: LG="1"                # amount of load generators SLAVE="1"             # amount of redis slaves TEST="1"              # 0 = Don't run tests, 1 = Do run tests. <del> NS="default"       # namespace </del> <ins> NS="k8petstore"       # namespace </ins> kubectl="${1:-$kubectl}" VERSION="${2:-$VERSION}"

During the review of , it's better to also update and to use non-default namespace. cc .
cc
/ , I can continue to work on this one :).
/ , can you help to assign this issue to me? So I can trace the tasks in my hand :).
I'd be happy to assign you once you accept the invitation to the org. Do you plan to work on this now? If you're not actively working on it, I'd prefer to leave it unassigned.
, would you help to send the invitation ? Yes, I'd like to work on this now, that's a good start for me to learn example & e2e test suite :).
Already sent. Visit to accept. See
, I'm kuber now , thanks. I'm working on this now :). Checked and , openshift-origin is not using default namespace now; and we can update 's script to update namespace. But it seems and other example are still using default namespace, should we go through all example and update them accordingly?
, should I update all ns in one commit or one commit per example?
In which examples do you plan to make this change? cc
Making this change in some examples would be useful, but only if we explain why we're doing it. Additionally, linking to existing docs and blogs about namespaces would be helpful: I don't want to do this in all of the examples, because it does add an extra step, and labels also provide a means of managing a group of related resources within a namespace. Perhaps could recommend examples where this would make sense.


Question: local attempt=0 while true; do <del> if ! aws s3 ls "s3://${AWS_S3_BUCKET}" > /dev/null 2>&1; then </del> <ins> if ! aws s3 ls --region ${AWS_S3_REGION} "s3://${AWS_S3_BUCKET}" > /dev/null 2>&1; then </ins> if (( attempt > 5 )); then echo echo -e "${color_red}Unable to confirm bucket creation." >&2

When I run: I get the following error: Waiting a bit and running got past this point because the bucket was ready the second time around, but the getting-started process should be more robust.
I experienced the same issue
We do try to query for the bucket existing, but perhaps S3 is even more eventually consistent than I thought! I did observe that we were very trusting about the user providing the correct S3 region, and have tried to fix this in . I don't think that was the problem here though (if you make_bucket, you can't have a region mismatch...) I guess we could retry the sync a few times; might help with e.g. a network interruption during the sync also.
I believe this to have been fixed by . I'm going to close, but if it happens again please do reopen.
FYI I got this with the latest k8s as of 10/13/15: I believe this is k8s 1.0.6 but I just tore down my AWS cluster and apparently kubectl needs to talk to the now-terminated master instance. I ran to get k8s. Waiting and running again doesn't show the issue since the bucket is fully created and accessible. I just provisioned a Vagrant cluster in hopes that would help show what version I'm using. Here's the output of :


Question: if _, err := os.Stat(rootPath); err != nil { if os.IsNotExist(err) { <del> return nil, fmt.Errorf("rootDirectory %q does not exist", rootPath) </del> <ins> if err := os.MkdirAll(path.Clean(rootPath), 0750); err != nil { return nil, fmt.Errorf("error creating root directory %q: %v", rootPath, err) } </ins> } else { return nil, fmt.Errorf("failed to Stat %q: %v", rootPath, err) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When running on a new VM, I get the following error: error: failed to run Kubelet: rootDirectory "/var/lib/kubelet" does not exist What you expected to happen: Previously, if this root directory did not exist, kubelet would automatically create it with the appropriate permissions. How to reproduce it (as minimally and precisely as possible): $ git clone $ cd kubernetes $ git checkout $ make WHAT=cmd/kubelet $ sudo ./output/dockerized/bin/linux/amd64/kubelet --fail-swap-on=false This invocation exits immediately with the above error about not existing. Anything else we need to know?: I believe this recently-merged PR is the cause: If I revert this PR's commit locally, and repeat the above steps, then gets automatically created when I start up kubelet: $ git revert $ make WHAT=cmd/kubelet $ sudo ./output/dockerized/bin/linux/amd64/kubelet --fail-swap-on=false This invocation stays running, and if I check in another window, has automatically been created. This discrepancy is currently breaking our . If the correct answer is that autocreation of the root directory was never supposed to be a supported feature and just happened to work, then we can update that expectation elsewhere to explicitly create the root directory first, but this does seem like a regression. Environment: Kubernetes version (use ): upstream/master HEAD (at ) Cloud provider or hardware configuration**: The same happens on my local desktop and a GCE VM with Ubuntu OS (e.g. from /etc/os-release): Ubuntu 14.04 on my desktop, 16.04 on GCE Kernel (e.g. ): 4.4.0-83-generic on my desktop, 4.4.0-21-generic on GCE Install tools: building/running directly from the source code using the above steps Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list .

The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
cc
/kind bug /area kubelet
/cc for
I am sure about the sequence of this? When cadvisor checks the directory, should this directory already exist?
figured out where exactly we create the directories. please see the PR
For default , we may create it if needed. But what if a customized value?
The behavior looks to be the same. There is a default root directory, but it can be overridden with a CLI flag. The code for directory creation is here, and uses the after CLI args have been resolved:
Thanks for the clarification. Actually . So we may remove the duplicated in . WDYT?
i am not comfortable with the duplicate from setupDataDirs as someone can use Kubelet directly without using the code under (and without using cadvisor). So we should let it be
SGTM. Thanks.


Question: return fmt.Errorf("unable to register configz: %s", err) } <del> return Run(cc, stopCh) </del> <ins> return Run(cc, stopCh, registryOptions...) </ins> } // Run executes the scheduler based on the given configuration. It only return on error or when stopCh is closed. <del> func Run(cc schedulerserverconfig.CompletedConfig, stopCh <-chan struct{}) error { </del> <ins> func Run(cc schedulerserverconfig.CompletedConfig, stopCh <-chan struct{}, registryOptions ...Option) error { </ins> // To help debugging, immediately log version klog.V(1).Infof("Starting Kubernetes Scheduler version %+v", version.Get()) <ins> registry := framework.NewRegistry() for _, option := range registryOptions { if err := option(registry); err != nil { return err } } </ins> // Create the scheduler. sched, err := scheduler.New(cc.Client, cc.InformerFactory.Core().V1().Nodes(),

<!-- Please only use this template for submitting enhancement requests --What would you like to be : As the is now implementable, it will be easier to add out of tree scheduler. And we are trying to use it to implement our plugins. The issue tracks following problems: Implement interface same as Custom Scheduler Plugins (out of tree)  to support out of tree scheduler Add support for scheduler to load and register user’s out of tree scheduling plugins that could be configured by admins. We could use to implement it, however it has some drawbacks as pointed out, e.g. We need find other ways, e.g. . /sig scheduling /assign
I did some investigation. It seems that is a better choice for the second part of this issue. I am making a prototype for it. Need I send a proposal at first? /cc
Just wrote a small proposal for it


Question: reason = inspectResult.State.Error } result.status.State.Termination = &api.ContainerStateTerminated{ <del> ExitCode:   inspectResult.State.ExitCode, Reason:     reason, StartedAt:  util.NewTime(inspectResult.State.StartedAt), FinishedAt: util.NewTime(inspectResult.State.FinishedAt), </del> <ins> ExitCode:    inspectResult.State.ExitCode, Reason:      reason, StartedAt:   util.NewTime(inspectResult.State.StartedAt), FinishedAt:  util.NewTime(inspectResult.State.FinishedAt), ContainerID: DockerPrefix + dockerID, </ins> } if tPath != "" { path, found := inspectResult.Volumes[tPath]

Currently kubelet reports current status only. If pod's restart policy is not never, the last terminated container status will be override by the latest status, and lost. We should report last terminated container status. I am going to update our v1beta3 api first, but will hold the implementation until we have
did you have a chance to start working on it? IIUC, to support this we need to change the api (), which IIUC from what wrote in some other thread should be done this week. I'm happy to help with it if you want me to help.
Only breaking changes have a deadline. Additive changes can be done at any time.
I am working on this. PR should be ready today or tomorrow.
Also agree with This is not part of API breakage.
Thanks for answers & clarifications and


Question: f.NamespacePodSecurityLevel = admissionapi.LevelBaseline ginkgo.Describe("CSI Conformance", func() { <del> ginkgo.It("should run through the lifecycle of a StorageClass", func(ctx context.Context) { </del> <ins> /* Release: v1.29 Testname: StorageClass, lifecycle Description: Creating a StorageClass MUST succeed. Reading the StorageClass MUST succeed. Patching the StorageClass MUST succeed with its new label found. Deleting the StorageClass MUST succeed and it MUST be confirmed. Replacement StorageClass MUST be created. Updating the StorageClass MUST succeed with its new label found. Deleting the StorageClass via deleteCollection MUST succeed and it MUST be confirmed. */ framework.ConformanceIt("should run through the lifecycle of a StorageClass", func(ctx context.Context) { </ins> scClient := f.ClientSet.StorageV1().StorageClasses() var initialSC, replacementSC *storagev1.StorageClass

[X] test approval issue : [X] test pr : [X] two weeks soak end date : 22 Sept 2023 [X] test promotion pr : [X] remove endpoint from According to following APIsnoop query, there are three StorageClass endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following StorageClass endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted /cc
/sig storage
/reopen For tracking the rest of the steps to Promotion
Reopened this issue. <detailsIn response to : /reopen For tracking the rest of the steps to Promotion Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: }}, }, }, rest.ValidateAllObjectFunc, &metav1.CreateOptions{}) <del> svc = obj.(*api.Service) </del> if err != nil { t.Fatalf("Expected no error: %v", err) } <ins> svc = obj.(*api.Service) </ins> _, _, err = storage.Update(ctx, svc.Name, rest.DefaultUpdatedObjectInfo(&api.Service{ ObjectMeta: metav1.ObjectMeta{ Name:            svc.Name,

Which test(s) are flaking: TestServiceRegistryUpdateDryRun Reason for failure: This is flaking rarely enough that it is not caught by our CI jobs, which currently tolerate up to 2 unit test failures per run (!). With that toleration removed in , this flake has been seen (). Reproducible with the following steps: the bug in the test that casts the return value even when the create fails: logging on slow etcd writes because of etcd-io/etcd a test binary for the affected package: the affected test using the tool: failures are seen, look at the logs to see the details about the error: /sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
heh this, was a funny one, we had 1/2^8 chances of using a duplicate, with 1/2^16 I've got 2000 runs without any failures
/assign /remove-triage unresolved


Question: # See the License for the specific language governing permissions and # limitations under the License. <ins> # This script generates `/pkg/kubelet/apis/podresources/v1alpha1/api.pb.go` # from the protobuf file `/pkg/kubelet/apis/podresources/v1alpha1/api.proto` # for pods. # Usage: `hack/update-generated-pod-resources.sh`. </ins> set -o errexit set -o nounset set -o pipefail

What would you like to be : There are many useful scripts in "". For example, I often use and for creating simple k8s cluster quickly. However, there are only few explanation or nothing how to use it. I think it becomes more useful and helpful if some instruction or guidance is explained in each script. Why is this needed: It aims and expects to increase new k8s developers by describing how to use each script. My proposal Add the following description under the copyright for each script file. Overview of the script Simple usage Supplementary explanation(This is option) Other concern I would like to know the purpose of use about hack directory. (e.g. Where to put tools for developers(therefore there is no guidance?).)
I don't know who is in charge of each scripts of hack, but if he agrees the idea of this feature request , I am going to do this task myself!!
/sig testing
I believe there is a script generating swagger/openapi specfilters under hack, but I am still struggling with it. It could be of great help if there is some how-to doc for each scripts.
this basis, why not create a file under the hack folder, because a lot of kk CIs are derived from multiple script commands under the hack. There are already 268+ files under the hack file. For contributors unfamiliar with these, these comments and instructions are helpful to them, and it is also helpful for new contributors to handle pr ci error reports.
/cc What do you think
I think it's good idea to create (It may be easy to understand if the script's explanation is described by category or use in ) Apart from that, it is helpful that the script file itself has a brief description.
When we create , it may tough work if creating it including all scripts explanation at one time. (There are many scripts and I understand only some scripts now.) Regarding , current my idea is to add a description to each script little by little and update the at the same time. This means that will be completed little by little in conjunction with adding the description of each script. In the future, if someone create new script in hack, he also may have to add description of his script into
Self-documenting commands sound useful. A high level overview also sounds useful.

/cc /cc Regarding adding help information, I suggest that we can submit pr in one or several files as a unit, so that we can track and statistics. For the documentation, I think it should be concise and not too long. How about this advice? We can claim tasks through , the pr title yes , i agree with you. [x] kubernetes/hack/benchmark- [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] kubernetes/hack/boilerplate/BUILD [ ] kubernetes/hack/boilerplate/test/BUILD [ ] [ ] [ ] [ ] [ ] kubernetes/hack/BUILD [x] kubernetes/hack/build- [x] kubernetes/hack/build- [x] [ ] kubernetes/hack/conformance/BUILD [ ] [x] kubernetes/hack/dev-build-and- [x] kubernetes/hack/dev-build-and- [x] kubernetes/hack/dev-push- [ ] kubernetes/hack/e2e-internal/BUILD [ ] kubernetes/hack/e2e-internal/e2e-cluster- [ ] kubernetes/hack/e2e-internal/e2e- [ ] kubernetes/hack/e2e-internal/e2e-grow- [ ] kubernetes/hack/e2e-internal/e2e-shrink- [ ] kubernetes/hack/e2e-internal/e2e- [ ] kubernetes/hack/e2e-internal/e2e- [x] kubernetes/hack/e2e-node- [ ] kubernetes/hack/generate- [x] kubernetes/hack/generate- [ ] kubernetes/hack/gen-swagger-doc/gen-swagger- [x] kubernetes/hack/get- [x] kubernetes/hack/ginkgo- [x] kubernetes/hack/grab- [x] kubernetes/hack/install- [ ] kubernetes/hack/jenkins/benchmark- [ ] kubernetes/hack/jenkins/OWNERS [ ] [ ] kubernetes/hack/jenkins/test- [ ] kubernetes/hack/jenkins/verify- [ ] kubernetes/hack/lib/BUILD [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [ ] [x] kubernetes/hack/lint- [x] kubernetes/hack/list-feature- [x] kubernetes/hack/local-up- [ ]

kubernetes/hack/make-rules/BUILD [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make-rules/helpers/go2make/BUILD [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with- [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/skipme/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/subdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/testdata/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/subsubdir/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/underscore/README [ ]

kubernetes/hack/make-rules/helpers/go2make/testdata/dir-with-gofiles/README [ ] kubernetes/hack/make-rules/helpers/go2make/testdata/dir-without-gofiles/README [ ] kubernetes/hack/make-rules/Makefile.manifest [ ] kubernetes/hack/make-rules/make- [ ] kubernetes/hack/make-rules/test- [ ] kubernetes/hack/make-rules/test-e2e- [ ] kubernetes/hack/make-rules/test-e2e- [x] kubernetes/hack/make-rules/test- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/make- [ ] kubernetes/hack/OWNERS [x] kubernetes/hack/pin- [x] kubernetes/hack/print-workspace- [x] kubernetes/hack/run-in- [ ] kubernetes/hack/testdata/auth- [x] [x] [x] kubernetes/hack/testdata/CRD/foo-- [x] kubernetes/hack/testdata/CRD/foo-deleted- [x] kubernetes/hack/testdata/CRD/foo-updated- [x] [x] kubernetes/hack/testdata/CRD/multi-crd-list-- [x] kubernetes/hack/testdata/CRD/multi-crd-list-deleted- [x] kubernetes/hack/testdata/CRD/multi-crd-list-updated- [x] kubernetes/hack/testdata/CRD/multi-crd- [x] [x] [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-label- [x] kubernetes/hack/testdata/deployment-multicontainer- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment- [x] kubernetes/hack/testdata/deployment-with- [x] [x] [x] [x] [x] [x] kubernetes/hack/testdata/filter/pod-apply- [x] kubernetes/hack/testdata/filter/pod-dont- [x]

kubernetes/hack/testdata/frontend- [x] kubernetes/hack/testdata/frontend- [x] kubernetes/hack/testdata/invalid-rc-with-empty- [x] kubernetes/hack/testdata/kubernetes- [x] [x] [x] [x] [x] [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-json- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-list- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-rclist- [ ] kubernetes/hack/testdata/multi-resource- [ ] kubernetes/hack/testdata/multi-resource-svclist- [x] kubernetes/hack/testdata/multi-resource-yaml- [x] kubernetes/hack/testdata/multi-resource- [x] kubernetes/hack/testdata/null-propagation/deployment- [x] kubernetes/hack/testdata/null-propagation/deployment- [x] kubernetes/hack/testdata/OWNERS [x] kubernetes/hack/testdata/pod- [x] kubernetes/hack/testdata/pod- [x] kubernetes/hack/testdata/pod-with-api- [x] kubernetes/hack/testdata/pod-with-large- [x] kubernetes/hack/testdata/pod-with- [x] [x] [x] [x] [x] kubernetes/hack/testdata/prune- [x] kubernetes/hack/testdata/prune- [x] kubernetes/hack/testdata/recursive/deployment/deployment/nginx- [x] [x] [x] [x] [x] kubernetes/hack/testdata/recursive/pod/pod/busybox- [x] [x] [x] kubernetes/hack/testdata/recursive/pod- [x] kubernetes/hack/testdata/recursive/pod-modify/pod/busybox- [x] kubernetes/hack/testdata/recursive/pod- [x] kubernetes/hack/testdata/recursive/pod- [x] [x] kubernetes/hack/testdata/recursive/rc/rc/busybox- [x] [x] [x] kubernetes/hack/testdata/redis-slave- [x]

kubernetes/hack/testdata/retainKeys/deployment/deployment- [x] kubernetes/hack/testdata/retainKeys/deployment/deployment- [x] kubernetes/hack/testdata/rollingupdate-daemonset- [x] kubernetes/hack/testdata/rollingupdate- [x] kubernetes/hack/testdata/rollingupdate-statefulset- [x] kubernetes/hack/testdata/rollingupdate- [x] kubernetes/hack/testdata/scale-deploy-1.yaml [x] kubernetes/hack/testdata/scale-deploy-2.yaml [x] kubernetes/hack/testdata/scale-deploy-3.yaml [x] [x] kubernetes/hack/testdata/service- [x] kubernetes/hack/testdata/service- [x] kubernetes/hack/testdata/sorted-pods/sorted- [x] kubernetes/hack/testdata/sorted-pods/sorted- [x] kubernetes/hack/testdata/sorted-pods/sorted- [ ] [ ] [x] kubernetes/hack/test- [ ] kubernetes/hack/test- [x] kubernetes/hack/update- [ ] kubernetes/hack/update- [x] kubernetes/hack/update- [ ] kubernetes/hack/update-generated-api-compatibility- [x] kubernetes/hack/update-generated-device-plugin- [ ] kubernetes/hack/update-generated-device- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-kms- [ ] kubernetes/hack/update-generated- [x] kubernetes/hack/update-generated-kubelet-plugin-registration- [ ] kubernetes/hack/update-generated-kubelet-plugin- [x] kubernetes/hack/update-generated-pod-resources- [ ] kubernetes/hack/update-generated-pod- [ ] kubernetes/hack/update-generated-protobuf- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-runtime- [ ] kubernetes/hack/update-generated- [ ] kubernetes/hack/update-generated-swagger- [x] kubernetes/hack/update- [x] kubernetes/hack/update-import- [ ] kubernetes/hack/update-openapi- [x] kubernetes/hack/update- [ ] kubernetes/hack/update-vendor- [ ] kubernetes/hack/update- [ ] kubernetes/hack/update-workspace- [x] kubernetes/hack/verify-

[x] kubernetes/hack/verify-api- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-cli- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-conformance- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-external-dependencies- [ ] kubernetes/hack/verify-flags/excluded- [x] kubernetes/hack/verify-flags/OWNERS [x] kubernetes/hack/OWNERS [ ] kubernetes/hack/verify-flags- [x] kubernetes/hack/verify-generated-device- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-files- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-kubelet-plugin- [x] kubernetes/hack/verify-generated-pod- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated- [x] kubernetes/hack/verify-generated-swagger- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-import- [x] kubernetes/hack/verify-import- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-no-vendor- [x] kubernetes/hack/verify-openapi- [x] kubernetes/hack/verify-pkg- [x] kubernetes/hack/verify-publishing- [x] kubernetes/hack/verify-readonly- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-staging-meta- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-test- [x] kubernetes/hack/verify-typecheck- [x] kubernetes/hack/verify- [x] kubernetes/hack/verify-vendor- [x]

kubernetes/hack/verify- [x]
i will pick up , i will do it step by step.
I agree with your idea, and thanks the checkbox to check progress. there are many scripts, so let's work together!
/assign /assign
I will pick up hack/verify-
I will pick up hack/verify-
I will pick up hack/verify- (all of verify scripts). But there are large number of scripts(over 40), I want to split it into two or three PRs.
I will pick up hack/update-
I will pick up these scripts. kubernetes/hack/benchmark- kubernetes/hack/build- kubernetes/hack/build- kubernetes/hack/dev-build-and- kubernetes/hack/dev-build-and- kubernetes/hack/dev-push- kubernetes/hack/e2e-node- kubernetes/hack/generate-
I will pick up these scripts get- ginkgo- grab- install- lint- list-feature- local-up- pin- print-workspace- run-in- test- test-
How about this now? /help Marking this tag so that others can easily access it.
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : How about this now? /help Marking this tag so that others can easily access it. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we've finished adding simple explanations to the almost scripts. It seems that several scripts of update- does not complete yet. I think we may able to close after that work.
I will pick up these scripts:
/assign


Question: resp.WriteHeader(http.StatusServiceUnavailable) } else { resp.WriteHeader(http.StatusOK) // In older releases, the returned "lastUpdated" time indicated the last // time the proxier sync loop ran, even if nothing had changed. To // preserve compatibility, we use the same semantics: the returned // lastUpdated value is "recent" if the server is healthy. The kube-proxy // metrics provide more detailed information. lastUpdated = currentTime } fmt.Fprintf(resp, `{"lastUpdated": %q,"currentTime": %q}`, lastUpdated, currentTime) }

We upgraded iptables on one of our hosts, and kube-proxy began logging errors about being unable to acquire the . However, we didn't notice this, because kube-proxy's health check () was still passing (and that's generally what we look at as our sign that something is amiss.) As a result, kube-proxy ran for days without being able to update iptables rules, eventually causing an outage for one of our services. We expected that kube-proxy's health check would fail if it was not able to interface with iptables. Our system was running Debian stretch, running kube-proxy in iptables mode with systemd, with iptables version 1.6.0. Given that starting configuration, the repro case is: iptables to 1.6.2. that kube-proxy begins emitting logs saying that it cannot update iptables because another process has the . that kube-proxy's health check endpoint () does not return an error. Example health check output: <details</details<details</details<detailsNone </details<details</details<detailsThe hard way </details<detailsDocker version 19.03.8, build </details<detailsCalico v3.18.4 </details
/sig network
you upgraded iptables just one node ? what is the version of iptables on other nodes ?
does it means that is not able to never get the lock or is just that sometimes is able to get it?
Yeah we upgraded on just one node at first. Like my original bug report says, we started at 1.6.0, so all the other nodes had 1.6.0. it never gets the lock. You have to restart kube-proxy so it can go through the iptables version detection routine again and change its behavior to the 1.6.2 logic branch. That's fine, it's whatever, but the health check doesn't fail even though kube-proxy can't do its job, which is a bug in my opinion. Also yes, we checked to see if some other process had the lock. It didn't. It's something to do with changes in how iptables actually works after the upgrade to 1.6.2, specifically. I believe this launchpad issue is relevant:
the logic of the healthcheck should fail if is not able to process the updates, right?

I this to the description as well, but here is an example response during this failure mode: And here are what the logs look like: Running gives no output.
This is interesting the and values are the same. This is the healthz logic, it will be unhealthy if the difference between both is greater than the health timeout. is it possible to share the entire kube-proxy log ? I want to discard something
to save me some time, could you please tell me what sort of messages you are looking for? I would rather go search splunk for them than to set up a repro case again.
just the first 1000 lines since kube-proxy starts should be enough
The JSON is kind of meaningless these days; if the proxier thinks it's healthy, it will just write out the current time as both and . It's only when it thinks its unhealthy that it writes out the real time. See I'm not sure how this could happen...
I'm afraid I'm not at liberty to give you 1000 lines of logs without some more targeted context. Company policy and whatnot. What I will tell you is I put the proxy up to and reproduced the issue and pulled those logs, and if you tell me the kind of messages you are looking for, I can pull the (redacted) context around them. What I can give you now, even without being directed, is the more verbose version of what I posted before, i.e., what happens around the time that the error gets logged: <detailsFollowed by the whole list of iptables rules. Then </detailsI can also give you the startup arguments: <details</detailsAnd finally, I can give you these lines from the iptables code, the health check getting registered, and a couple other startup items: <details</detailsAny more than this, though, and I'll have to ask you to tell me something to grep for.
First thing to check is that the healthcheck actually works EVER. Given the same timestamp (we should fix that?) it thinks it is healthy.
/assign
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
A situation: when there is a service or endpoint is constantly updated (which will update the ): so there is a high probability that the difference between the and the is less than the , so this will be considered healthy If you reduce ( iptables-sync-period * 2, the defaults value of flag is 30s), you may see response code is 503, but in fact, it is not easy to estimate how much it should be reduced. so As for case 3, I think there are something that can be optimized.
Oh! That's wrong. We don't care about the time of the most-recently queued update, we care about the time of the oldest queued update that was after the last . So rather than having and updating it every time gets called, we should have a variable something like , and can set it to 0, and should only set it to if it was previously 0. In fact, we don't even need . Kube-proxy would be healthy if or .
Thanks I see, I'd fix it, can I take this one?
sure /assign


Question: ginkgo.By(fmt.Sprintf("deleting the test namespace: %s", ns1)) // Delete the primary namespace but its okay to fail here because this namespace will // also be deleted by framework.Aftereach hook <del> tryFunc(deleteNamespaceFunc(f.ClientSet, ns1, framework.DefaultNamespaceDeletionTimeout)) </del> <ins> tryFunc(func() { f.DeleteNamespace(ns1) }) </ins> ginkgo.By("uninstalling csi mock driver") tryFunc(cleanup) tryFunc(cancelLogging) ginkgo.By(fmt.Sprintf("deleting the driver namespace: %s", ns2)) <del> tryFunc(deleteNamespaceFunc(f.ClientSet, ns2, framework.DefaultNamespaceDeletionTimeout)) </del> <ins> tryFunc(func() { f.DeleteNamespace(ns2) }) </ins> // cleanup function has already ran and hence we don't need to run it again. // We do this as very last action because in-case defer(or AfterEach) races // with AfterSuite and test routine gets killed then this block still

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: ci-kubernetes-e2e-ubuntu-gce-containerd Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Pre-provisioned PV (default fs)] subPath should support file as subpath [LinuxOnly] Testgrid link: Reason for failure: Will look at it shortly, why it is flaking. Anything else we need to know: links to appreciatedlinks to specific failures in spyglass appreciated
/sig storage
The namespace terminating during test setup is super strange but seems like a test-only issue, not an issue with the functionality being tested. I'd recommend looking at the test, API server, and controller logs to figure out why the test namespace () was deleted and who deleted it
does this help ?
feels like a test only problem like said. We don't even have gluster in our CI/prow to test against... cc Michelle, Xing, Can you please confirm that this is not a blocker for 1.19.0?
Knowing the timeframe for when the test started and when that delete happened would be interesting to know. Possibilities I can see: something in the test timed out, and the cleanup happened in parallel with something in the test attempting to run multiple tests are reusing the same test context and namespace name at the same time (I can never keep straight how ginkgo parallel tests work)
This does not look like a blocker for 1.19.0.
Thanks /milestone v1.20
looking at the logs it seems to be the latter "same test context and namespace name at the same time", but we can deal with this for 1.19.1
cc This looks like another instance of "the same test namespace is being reused". I believe had a mitigation for that to try to make the namespaces more unique.
(btw, the test brings up a containerized gluster server so we are exercising gluster functionality)
Yeah, looking at the build log found that it is the same issue as The issue is still around the double deletion of namespace, i.e. we delete the test namespace during the test execution, but later on the call we delete the namespace again to ensure it truly get cleaned up.

But unfortunately, this namespace has been reused by another test case between this gap.
Some potential fix I can think of: deletion of the primary namespace twice is inevitable, maybe we should consider generate primary namespace based on a hashing of the test cases name? the 4 digits random namespace to 6, this should not solve the problem in nature but it will extremely decrease the chance it happens. using an in-memory namespace map to keep record of already generated namespaces for the whole test run and refuse to generate the same namespace again in the whole lifecycle of a prow job.
do you know if this gluster test is using your mitigation in ?
hash of the testcase name + random suffix (because tests can be run multiple times) seems like a good way to mitigate the chances of tests reusing the same namespace. I'm unsure if we know the test case name at the time of test namespace creation though. I think increasing the length of the random suffix is a quick and easy mitigation believe ginkgo to run in parallel, so this may not work
Yes it is. But my fix in can only reduce the chance of collision and probably improve readability of logs. But not to eliminate this issue in nature.
AFAICT does not affect intree code at all, because it only fixes namespace of driver. If we are seeing collision of primary namespace for intree code then it is a different problem.
on only deals with the collision between test namespace and driver namespace. And that fix is taking affect. failure is because a test namespace collides with another test namespace. And this is not related to in-tree code at all. Any test could potentially become flaky because their namespace can be deleted during test execution. It's just happens to be this intree test this time. The root cause for this problem is: We cleaned up the test namespace twice which gives it a small gap that another testcase can generate the same namespace and be affected by the second namespace deletion from the original testcase.
/assign I would like to give it a shot.
I've opened to address this issue.
/unassign /assign


Question: waitForAttachTimeout time.Duration, volumeToMount VolumeToMount, actualStateOfWorld ActualStateOfWorldMounterUpdater, <del> isRemount bool) (volumetypes.GeneratedOperations, error) { </del> <ins> isRemount bool) volumetypes.GeneratedOperations { </ins> // Get mounter plugin <ins> volumePluginName := unknownVolumePlugin </ins> volumePlugin, err := og.volumePluginMgr.FindPluginBySpec(volumeToMount.VolumeSpec) <del> if err != nil || volumePlugin == nil { return volumetypes.GeneratedOperations{}, volumeToMount.GenerateErrorDetailed("MountVolume.FindPluginBySpec failed", err) </del> <ins> if err == nil && volumePlugin != nil { volumePluginName = volumePlugin.GetPluginName() </ins> } <del> affinityErr := checkNodeAffinity(og, volumeToMount, volumePlugin) if affinityErr != nil { eventErr, detailedErr := volumeToMount.GenerateError("MountVolume.NodeAffinity check failed", affinityErr) og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeWarning, kevents.FailedMountVolume, eventErr.Error()) return volumetypes.GeneratedOperations{}, detailedErr } </del> <ins> mountVolumeFunc := func() (error, error) { if err != nil || volumePlugin == nil { return volumeToMount.GenerateError("MountVolume.FindPluginBySpec failed", err) } </ins> <del> volumeMounter, newMounterErr := volumePlugin.NewMounter( volumeToMount.VolumeSpec, volumeToMount.Pod, volume.VolumeOptions{}) if newMounterErr != nil { eventErr, detailedErr := volumeToMount.GenerateError("MountVolume.NewMounter initialization failed", newMounterErr) og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeWarning, kevents.FailedMountVolume, eventErr.Error()) return volumetypes.GeneratedOperations{}, detailedErr } </del> <ins> affinityErr := checkNodeAffinity(og, volumeToMount, volumePlugin) if affinityErr != nil { return volumeToMount.GenerateError("MountVolume.NodeAffinity check failed", affinityErr) } </ins> <del> mountCheckError := checkMountOptionSupport(og, volumeToMount, volumePlugin) </del> <ins> volumeMounter, newMounterErr := volumePlugin.NewMounter( volumeToMount.VolumeSpec, volumeToMount.Pod, volume.VolumeOptions{}) if newMounterErr != nil { return volumeToMount.GenerateError("MountVolume.NewMounter initialization failed", newMounterErr) </ins> <del> if mountCheckError != nil { eventErr, detailedErr := volumeToMount.GenerateError("MountVolume.MountOptionSupport check failed", mountCheckError) og.recorder.Eventf(volumeToMount.Pod, v1.EventTypeWarning, kevents.UnsupportedMountOption, eventErr.Error()) return volumetypes.GeneratedOperations{}, detailedErr } </del> <ins> } </ins> <del> // Get attacher, if possible attachableVolumePlugin, _ := og.volumePluginMgr.FindAttachablePluginBySpec(volumeToMount.VolumeSpec) var volumeAttacher volume.Attacher if attachableVolumePlugin != nil { volumeAttacher, _ = attachableVolumePlugin.NewAttacher() } </del> <ins> mountCheckError := checkMountOptionSupport(og, volumeToMount, volumePlugin) </ins> <del> // get deviceMounter, if possible deviceMountableVolumePlugin, _ := og.volumePluginMgr.FindDeviceMountablePluginBySpec(volumeToMount.VolumeSpec) var volumeDeviceMounter volume.DeviceMounter if deviceMountableVolumePlugin != nil { volumeDeviceMounter, _ = deviceMountableVolumePlugin.NewDeviceMounter() } </del> <ins> if mountCheckError != nil { return volumeToMount.GenerateError("MountVolume.MountOptionSupport check failed", mountCheckError) } </ins> <del> var fsGroup *int64 if volumeToMount.Pod.Spec.SecurityContext != nil && volumeToMount.Pod.Spec.SecurityContext.FSGroup != nil { fsGroup = volumeToMount.Pod.Spec.SecurityContext.FSGroup } </del> <ins> // Get attacher, if possible attachableVolumePlugin, _ := og.volumePluginMgr.FindAttachablePluginBySpec(volumeToMount.VolumeSpec) var volumeAttacher volume.Attacher if attachableVolumePlugin != nil { volumeAttacher, _ = attachableVolumePlugin.NewAttacher() } // get deviceMounter, if possible deviceMountableVolumePlugin, _ := og.volumePluginMgr.FindDeviceMountablePluginBySpec(volumeToMount.VolumeSpec) var volumeDeviceMounter volume.DeviceMounter if deviceMountableVolumePlugin != nil { volumeDeviceMounter, _ = deviceMountableVolumePlugin.NewDeviceMounter() } var fsGroup *int64 if volumeToMount.Pod.Spec.SecurityContext != nil && volumeToMount.Pod.Spec.SecurityContext.FSGroup != nil { fsGroup = volumeToMount.Pod.Spec.SecurityContext.FSGroup } </ins> <del> mountVolumeFunc := func() (error, error) { </del> devicePath := volumeToMount.DevicePath if volumeAttacher != nil { // Wait for attachable volumes to finish attaching

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In this test, volume mount failed because the driver-registrar took a minute to come up. However, it seems like volume mount operation is retrying without any sort of backoff. This message appears in 400+ times in less than a minute. What you expected to happen: Operation executor should not be retrying operations so frequently. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'll take a look at this. /assign
Problem is that reconciler depends on operation executor to do backoff. In the normal case operation executor returns a method that does the actual work. Any errors by that method result in exponential backoff. But in this case, the error is coming from the code creating the mount method which is not handled by the back off logic, and is called again and again by the reconciler resulting in log and event spam. This didn't use to be a problem because initialization errors were uncommon (plugins were built in), but with CSI the driver may not exist and these initialization errors are expected. I'll look in to cleaning this up.
Is this something that will affect users in 1.13 and needs CPing?
This will result in kubelet log spam and possibly kubernetes event spam. It happens when someone creates a pod referencing a CSI volume before the CSI driver for that volume is installed. Which happens frequently in E2E tests because they install/remove driver with every test. But should be rare in real world usage since most users are likely to install a driver first before using it.
If release team is worried about this, we could create a quick PR to suppress logging/event generation of this these errors. Otherwise we plan to address this more holistically for 1.14 and 1.13.1.


Question: klog.V(4).InfoS("Cannot get host IPs", "err", err) } else { s.HostIP = hostIPs[0].String() <del> if kubecontainer.IsHostNetworkPod(pod) && s.PodIP == "" { s.PodIP = hostIPs[0].String() s.PodIPs = []v1.PodIP{{IP: s.PodIP}} if len(hostIPs) == 2 { </del> <ins> // HostNetwork Pods inherit the node IPs as PodIPs. They are immutable once set, // other than that if the node becomes dual-stack, we add the secondary IP. if kubecontainer.IsHostNetworkPod(pod) { // Primary IP is not set if s.PodIP == "" { s.PodIP = hostIPs[0].String() s.PodIPs = []v1.PodIP{{IP: s.PodIP}} } // Secondary IP is not set #105320 if len(hostIPs) == 2 && len(s.PodIPs) == 1 { </ins> s.PodIPs = append(s.PodIPs, v1.PodIP{IP: hostIPs[1].String()}) } }

What happened: We have installed k8s cluster on openstack cloud provider setup. K8s cluster is dual-stack enabled. During fresh deployment or worker node reboots (master nodes , worker nodes) , k8s control plane pods (other host networking pods aswell) such as kube-proxy, kube-scheduler , kube-controller-manager get only 1 IP address instead of dual-stack address. What you expected to happen: We expect all host networking pods to get dual-stack addresses , since cluster is dual-stack enabled. How to reproduce it (as minimally and precisely as possible): a) Openstack cloud provider setup with minimum 4-5 VMs. b) 3 master nodes, 2 worker nodes c) Execute simultaneous reboot of all nodes. Environment: Kubernetes version : v1.22.0
/sig node
/sig network /triage needs-information /assign Hi When you describe your node what are the IPs assigned in node object? kubelet copies from node object into pods. The way they are set on node object is by the cloud provider, manual via kubelet's flag or auto-detect which prefers IPv4 (check: ).
when i describe the node object , they contain dual-stack addresses. Yes i debugged kubelet with few additional logs. I could see that when kubelet is reading node obj to create these static pods, it sees only 1 IP. But eventually node object is populated with both the addresses. I also understand, is for baremetal installation. But we have cloud provider setup which might be delaying setting both the IPs in node obj. Before cloud provider sets both the IPs ,kubelet is fast enough to create these static pods. Let me know if you need any additional logs.
thanks again Few additional questions: are these static pods? When the node object gets its dualstack addresses and pods were created after the node object updates do you still get single address pods?
I've chatted with the reporter in slack, it seems a race on populating the node addresses by the cloud-provider,
Yes. It would seem so. Two things here i am trying to clarify. cloud providers node register with taints (no schedule) until cloud provider adds provider-Id and other details including IPs.

these are static pods then I am not sure if kubelet respects no-schedule (those are respected by scheduler which is out of the picture for static pods) If they are then they will run - most likely - before node gets IP addresses from cloud provider. The questions i asked above should put some lights on this.
Yes i am referring to kube-proxy , kube-api-server ,kube-scheduler pods which use host-networking. This is not under our control. I mean , we are just rebooting the nodes in quick succession and at the end we just check the IP addresses of the above static pods. As i see the code , kubelet brings them up as soon as he is UP.
IMO the cloud provider really needs to figure out the IPs before kubelet starts. OR the node startup needs to block until both IP addresses are available so that the cloud provider and/or kubelet can read them.
Should we accept this as a bug? Do we think we need to force some taint on new nodes that cloud-providers need to clear (obviously not compatible, but you get the idea)?
-- There is a no-schedule taint already there (if cloud provider is configured on kubelet). It is a bug (or undocumented behavior). I think we should always update PodIPs in host network mode. /triage accepted
xref
so i looked at this again. I believe should fix this issue as well. i don't think we should place this behind a feature gate. We already update when the host IP changes. We just need to edit the release note of to describe the behavior and caveats. Thoughts?
Please ignore that comment. The PR does not solve for
/remove-sig node
: Any more info needed from me ?
no thank you. We are addressing this by and the related KEP.
, any idea when will the fix be available ?


Question: Per pod IP configuration is provider specific, so when making networking changes, its important to sand-box these as all providers may not use the same mechanisms (iptables, openvswitch, etc.) We should define a grains.conf key that captures more specifically what network configuration environment is being used to avoid future confusion across providers. <ins> ## Further reading The [cluster/saltbase](../cluster/saltbase) tree has more details on the current SaltStack configuration. </ins>

We need to document all the configuration derived by Salt. Ideally this would be a markdown doc that we kept as live documentation within the project. The main reasons to bother: It's hard right now for anyone approaching the directory to get a big picture view of what's is going on, especially across providers. If we ever want to do something disciplined w.r.t. upgrade between (non-dev) versions, we're probably going to need a place to look for the state of the world. Decreasing the velocity with which people can add to Salt configuration will encourage people to delete more Salt configuration. cc


Question: } } <del> func TestValidationError(t *testing.T) { </del> <ins> func TestValidationErrorUsefulMessage(t *testing.T) { </ins> s := NewFieldInvalid("foo", "bar", "deet").Error() <del> if !strings.Contains(s, "foo") || !strings.Contains(s, "bar") || !strings.Contains(s, "deet") || !strings.Contains(s, ValueOf(ValidationErrorTypeInvalid)) { t.Errorf("error message did not contain expected values, got %s", s) </del> <ins> t.Logf("message: %v", s) for _, part := range []string{"foo", "bar", "deet", ValidationErrorTypeInvalid.String()} { if !strings.Contains(s, part) { t.Errorf("error message did not contain expected part '%v'", part) } } type complicated struct { Baz   int Qux   string Inner interface{} KV    map[string]int } s = NewFieldRequired( "foo", &complicated{ Baz:   1, Qux:   "aoeu", Inner: &complicated{Qux: "asdf"}, KV:    map[string]int{"Billy": 2}, }, ).Error() t.Logf("message: %v", s) for _, part := range []string{ "foo", ValidationErrorTypeRequired.String(), "Baz", "Qux", "Inner", "KV", "1", "aoeu", "asdf", "Billy", "2", } { if !strings.Contains(s, part) { t.Errorf("error message did not contain expected part '%v'", part) } </ins> } }

We now have pretty structured way of validating api object, but the error message to end user is not descriptive, e.g. when creating a replication controller with restart policy 'never', we get: even as simple as label name, we shouldn't assume end-user is familiar with our requirement. Apart from what is wrong, we'd better also give why it is wrong.
The main thing wrong here is not having a useful String() method. and have both been pretty adamant about not attaching code to API objects, but here's a clear UX justification. What if we only attached methods to internal API objects, not versioned ones? Also: this should print %#v, rather than %+v, which would be a BIT more readable.
Orrrr we could switch to spew, which is already a dep.
Actually we're printing a reflect.Value there, which makes no sense at all. I will fix. EDIT: no we aren't, I can't read-- will still fix though


Question: func ValidateNetworkPolicyUpdate(update, old *networking.NetworkPolicy) field.ErrorList { allErrs := field.ErrorList{} allErrs = append(allErrs, apivalidation.ValidateObjectMetaUpdate(&update.ObjectMeta, &old.ObjectMeta, field.NewPath("metadata"))...) <del> if !reflect.DeepEqual(update.Spec, old.Spec) { allErrs = append(allErrs, field.Forbidden(field.NewPath("spec"), "updates to networkpolicy spec are forbidden.")) } </del> <ins> allErrs = append(allErrs, ValidateNetworkPolicySpec(&update.Spec, field.NewPath("spec"))...) </ins> return allErrs }

I tried to update the NetworkPolicy Spec field with more ingress rules but got It seems NetworkPolicy does not allow update on Spec This means if I want to have more ingress rules, I will have to add those in a new networkpolicy in the same namespace. Any reason why I should not add those in the existing one(update the existing networkpolicy)? Thanks!
This is as designed for now. The expectation is that updates to the spec will be allowed when the NetworkPolicy API moves from v1beta1 to GA.
(And that is planned for Kubernetes 1.6 release.)
good to know, thanks!
Note that the PR to move NetworkPolicy to v1 did not make it into v1.6, but is now planned for 1.7:
Oh, I didn't know about this bug and that didn't get changed in but we can do it as another PR after. (I'm trying to not make any more changes there so we can get it merged.)


Question: deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", <ins> "//staging/src/k8s.io/apiserver/pkg/endpoints/metrics:go_default_library", </ins> "//staging/src/k8s.io/apiserver/pkg/server/httplog:go_default_library", "//vendor/k8s.io/klog:go_default_library", ],

What would you like to be : Currently the apiserver has request latencies metrics, which are a histogram of requests to the apiserver. However, these request metrics do not include healthz. While we have recently the ability to get back health check metrics from the healthz endpoint directly, we should also emit this data as a metric, so that we can monitor health check latencies. /assign /sig api-machinery instrumentation
GitHub didn't allow me to assign the following users: jktomer. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /sig api-machinery instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I noticed this is informally assigned. Has there been any progress? Is this a place I can fit in?
Yes, I'm working on a PR.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: redisImage    = imageutils.GetE2EImage(imageutils.Redis) nginxImage    = imageutils.GetE2EImage(imageutils.Nginx) busyboxImage  = imageutils.GetE2EImage(imageutils.BusyBox) <ins> agnhostImage  = imageutils.GetE2EImage(imageutils.Agnhost) </ins> ) var (

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: Which test(s) are failing: Kubectl client [] Kubectl logs should be able to retrieve and filter logs [Conformance] Since when has it been failing: Testgrid link: Reason for failure: . . switched to using the docker official redis image and the test string was changed accordingly. On Windows, we still use the old image with the old string, and this caused the test to fail. /cc
Hmm, there doesn't seem to be a redis port for windows 3.2.100 : I guess we can modify the test to find "ready to accept connections" rather that "Ready ... ".
/cc


Question: <del> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "net/http" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/golang/glog" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) func TestNetwork(c *client.Client) bool { if testContext.provider == "vagrant" { glog.Infof("Skipping test which is broken for vagrant (See https://github.com/GoogleCloudPlatform/kubernetes/issues/3580)") return true } // Test basic external connectivity. resp, err := http.Get("http://google.com/") if err != nil { glog.Errorf("unable to talk to the external internet: %v", err) return false } if resp.StatusCode != http.StatusOK { glog.Errorf("unexpected error code. expected 200, got: %v (%v)", resp.StatusCode, resp) return false } ns := api.NamespaceDefault // TODO(satnam6502): Replace call of randomSuffix with call to NewUUID when service //                   names have the same form as pod and replication controller names. name := "nettest-" + randomSuffix() svc, err := c.Services(ns).Create(&api.Service{ ObjectMeta: api.ObjectMeta{ Name: name, Labels: map[string]string{ "name": name, }, }, Spec: api.ServiceSpec{ Port:          8080, ContainerPort: util.NewIntOrStringFromInt(8080), Selector: map[string]string{ "name": name, }, }, }) glog.Infof("Creating service with name %s", svc.Name) if err != nil { glog.Errorf("unable to create test service %s: %v", svc.Name, err) return false } // Clean up service defer func() { if err = c.Services(ns).Delete(svc.Name); err != nil { glog.Errorf("unable to delete svc %v: %v", svc.Name, err) } }() rc, err := c.ReplicationControllers(ns).Create(&api.ReplicationController{ ObjectMeta: api.ObjectMeta{ Name: name, Labels: map[string]string{ "name": name, }, }, Spec: api.ReplicationControllerSpec{ Replicas: 8, Selector: map[string]string{ "name": name, }, Template: &api.PodTemplateSpec{ ObjectMeta: api.ObjectMeta{ Labels: map[string]string{"name": name}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:    "webserver", Image:   "kubernetes/nettest:latest", Command: []string{"-service=" + name}, Ports:   []api.Port{{ContainerPort: 8080}}, }, }, }, }, }, }) if err != nil { glog.Errorf("unable to create test rc: %v", err) return false } // Clean up rc defer func() { rc.Spec.Replicas = 0 rc, err = c.ReplicationControllers(ns).Update(rc) if err != nil { glog.Errorf("unable to modify replica count for rc %v: %v", rc.Name, err) return } if err = c.ReplicationControllers(ns).Delete(rc.Name); err != nil { glog.Errorf("unable to delete rc %v: %v", rc.Name, err) } }() const maxAttempts = 60 for i := 0; i < maxAttempts; i++ { time.Sleep(2 * time.Second) body, err := c.Get().Prefix("proxy").Resource("services").Name(svc.Name).Suffix("status").Do().Raw() if err != nil { glog.Infof("Attempt %v/%v: service/pod still starting. (error: '%v')", i, maxAttempts, err) continue } switch string(body) { case "pass": glog.Infof("Passed on attempt %v. Cleaning up.", i) return true case "running": glog.Infof("Attempt %v/%v: test still running", i, maxAttempts) case "fail": if body, err := c.Get().Prefix("proxy").Resource("services").Name(svc.Name).Suffix("read").Do().Raw(); err != nil { glog.Infof("Failed on attempt %v. Cleaning up. Error reading details: %v", i, err) } else { glog.Infof("Failed on attempt %v. Cleaning up. Details:n%v", i, string(body)) } return false } } if body, err := c.Get().Prefix("proxy").Resource("services").Name(svc.Name).Suffix("read").Do().Raw(); err != nil { glog.Infof("Timed out. Cleaning up. Error reading details: %v", err) } else { glog.Infof("Timed out. Cleaning up. Details:n%v", string(body)) } return false } var _ = Describe("TestNetwork", func() { It("should pass", func() { // TODO: Instead of OrDie, client should Fail the test if there's a problem. // In general tests should Fail() instead of glog.Fatalf(). Expect(TestNetwork(loadClientOrDie())).To(BeTrue()) }) }) </del>

Create a ginkgo networking test suite and add the following tests to it: TestNetwork TestImportantURLs
Part of the ginkgo-ization of the e2e tests


Question: componentstatuses=$(echo "${cs_status}" | grep -c 'Healthy:') || true healthy=$(echo "${cs_status}" | grep -c 'Healthy:True') || true <del> if ((componentstatuses > healthy)); then </del> <ins> if ((componentstatuses > healthy)) || ((componentstatuses == 0)); then </ins> if ((attempt < 5)); then echo -e "${color_yellow}Cluster not working yet.${color_norm}" attempt=$((attempt+1))

cluster upgrade should maintain a functioning cluster [Feature:ClusterUpgrade] connection refused in version check /sig networking /assign See also
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig network
I took a quickly look at the failures, the root cause is actually the master node is not ready. The following error msg indicates why master is not ready: It is indeed a network related issue.
I am re-assigning this back to you, please delegate. Thanks! cc/
this looks like a CNI related, can you guys take a look?
/assign
One thing I see happening at least in the GCE tests (Because it has the master log, but the GKE tests do not) I0614 04:25:10. 5 ] Node bootstrap-e2e-minion-group-2cz0 has CIDR 10.180.1.0/24, occupying it in CIDR map E0614 04:25:10. 5 ] Error starting "node" F0614 04:25:10. 5 ] error starting controllers: failed to mark cidr as occupied: cidr 10.180.1.0/24 is out the range of cluster cidr 10.100.0.0/14 I see that the test has: cluster/gce/config-"${CLUSTERIPRANGE:-10.100.0.0/14}" And this for the the pod cidr: "--pod-cidr", "10.180.0.0/24", // Assign a fixed CIDR to the node because there is no node controller. Is the above 10.100.0.0/14 supposed to be 10.180/14 ? 10.100/14 is only going to give 10.100 to 10.103 which isn't within the cluster cidr range.
I think I have an idea what happened here.

It looks like this PR changed the cluster CIDR from 10.180/14 to 10.100/14 However I see the the e2e node test assumes the pod cider is 10.180.0/24 Here: And Here: Since I believe we need to keep the 10.100/14 probably the right thing to do now is to fix the e2e node test to pick a pod cidr within 10.100/14
This was green on 6-15, if green again today i'll close this. Thanks for your attention !
should be the one fixing the issue. I will leave this open, until all UpgradeTest on GCE 1.6-1.7 are green. Thanks!
Ok, I thought we still have the issue on this, and assign this to myself. It turns out the test is still running with the old build for the upgrade test. Will continue monitoring this. cc/
fyi
After was merged, the master is up and ready, but the upgrade is still marked as failed. I looked into it this morning, and found controller_manager was running into the crashloop after master was upgraded due to the following error: I believe the issue is that we changed validation test in at which breaks the backward compatibility with 1.6 nodes. cc/ WDYT?
Had a quick discussion with network team The quick fix can be putting all nodes' cidr back 10.180.0.0/24, instead of 10.100.0.0/14 to stop the upgrade craziness. is working on this, and running the upgrade tests locally. Meanwhile, is going to discuss with on the next steps on how to rollout the feature which depends on the new ip range: 10.100.0.0/14
I think the problem now is that we have one version using 10.180.x. (1.6) and one using 10.100x.x. (1.7) And when we upgrade happens the cluster was created with 10.180 but then we upgrade we want 10.100. Can we just set the env var CLUSTERIPRANGE to 10.100 in the test so that the 1.6 cluster comes up with the right range?
In particular I think we could just change this file? To include the CLUSTERIPRANGE.

What you suggested should work theoretically. But since both branches hard-coded with two different ip range in some files, instead of respecting CLUSTERIPRANGE env variable, it won't work now. Since the failure are limited to our tests only, and upgrade test itself relies on the previous branch's latest build, not the real release, we decided to cherry pick fix to 1.6 branch. I closed my revert prs.
Cherrypicking the required changes to 1.6, 1.5:
besides cherrypicking , I believe we also need cherrypick which changes the API validation code; Otherwise, you are going to break 1.6 branch validation path on this. EDITED: Ignore this comment. All changes in are either for node e2e tests, or for validation test itself.
Thanks, looking forward to a green run :) Latest failure (in 1.6 -17 upgrade tests) is as in opening comment:
After cherrypicking those ip_range changes to 1.6 branch, now the master node is successfully upgraded to 1.7, and still accepted nodes running 1.6. The kube-controller-manager is not in crashloop anymore. The below is the output by running kubectl get nodes on the master node with https://k8s-
Also all kube-system pods are running successfully after master is upgraded:
Testing is still running, but I noticed that the cluster upgrade failed in the log at https://k8s- From above comment at , one can see that 1) now master node are up and ready, vs. notready in pre-54 runs 2) kube-controller-manager is running for 3 hour, not in crashloop in pre-54 runs kube-controller-manager-bootstrap-e2e-master 1/1 Running 3 3h 3) kubectl cs returns ok without error. There is no 'kubectl get nodes' failure, no kubect get cs failure. Above error / failure is red-herring, and looks like an autograde test infra issue. cc/
It looks like failed from the e2e test. I don't doubt that it worked from the master, but that's not where the e2e is running. I suspect an auth issue.

When the suite finishes, we can confirm by looking at the api server log for component status access from the kubectl user agent.
I checked kube-, there is no failure from the kubectl user agent, especially right before test claimed UpgradeMaster is failed, /componentstatus from kubectl is ok: I0620 00:23:45. 5 ] GET /api/v1/componentstatuses: (757.782364ms) 200 [[kubectl/v1.7.0 (linux/amd64) kubernetes/] 35.188.93.146:] In test log: https://k8s-, before previous /componentstatuses call, master is upgraded successfully, and all work-nodes are ready: But still complaint cluster is not working: But there is no retry with kubectl get cs which should be called multiple times in validate- and I checked validate- and think there is a problem in the code. The weirdness is that validation- hasn't been touched for a year. Something underneath triggers this change? is sending a quick fix.
Ok, I have more data to say that (along with ) is really buggy. Without any change, the new run of both master and nodes are successfully upgraded to 1.7 in a new run: https://k8s-
This guy is flakey: , Most recent failure about 11pm yesterday, still waiting for another run. Since the test is still failing, this issue should not have been closed - but the bot did it and don't have the authority to re-open. we're skipping some tests because of known API skew failures. Do you want to mark this test as a skip?
/reopen
Let's mark this as flakey and skip it.
we cannot skip this because it is does the real upgrade. If master upgrade failed, the test is red; if the node upgrade failed, the test is red. Sometimes, the script is flaky at the end, the test is red even the entire cluster is upgraded successfully. But we should skip this from 1.5-1.7 upgrade test suites since we only care about kubectl skews tests. There is no need to upgrade node from 1.5-1.7.
So we're going to ignore this in the 15-1.7 tests. Focus on the 1.6-1.7 tests.

Seeing new red in these (after some green) These guys remained green:
Is this related the aggregation API server ready issue?
What's the status of this issue? Should it continue blocking 1.7? If not, please remove the 'approved-for-milestone' label.
is on PTO
After we fixed the subnetwork issue, now upgrade are fine: https://k8s- There is failure for ingress And the engineer indicates it is caused by the infrastructure underneath.
I remove this as the release blocker.
This is failing in gce-1.7-master-upgrade-master: example job: https://k8s- I will apply release blocker and urgency to this. Please advice if it is not the case.
This did occur there, but it doesn't seem to occur often (hasn't happened in that suite since). The major issue with the suite that you linked is the panic you fixed yesterday.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: est.lock.Lock() defer est.lock.Unlock() <del> rrv := est.relatedResourceVersions(endpointSlice) </del> <ins> rrv, ok := est.relatedResourceVersions(endpointSlice) if !ok { return true } </ins> return rrv[endpointSlice.Name] != endpointSlice.ResourceVersion }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The endpointSliceTracker creates a set of resource versions for each service, the resource versions in the set could be deleted when endpointslices are deleted, but the set and its key in the map is never deleted, leading to memory leak. What you expected to happen: The set of resource versions should be delete from endpointSliceTracker when the service is deleted. Environment: Kubernetes version (use ): 1.18.5
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc


Question: rc.Status.AvailableReplicas == newStatus.AvailableReplicas && rc.Generation == rc.Status.ObservedGeneration && reflect.DeepEqual(rc.Status.Conditions, newStatus.Conditions) { <del> return nil </del> <ins> return &rc, nil </ins> } // Save the generation number we acted on, otherwise we might wrongfully indicate // that we've seen a spec update when we retry.

Is this a request for help? No. What keywords did you search in Kubernetes issues before filing this one? dangling ReplicaSet Is this a BUG REPORT or FEATURE REQUEST? BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure A1 OS (e.g. from /etc/os-release): NAME="Container Linux by CoreOS" ID=coreos VERSION=1235.9.0 VERSIONID=1235.9.0 BUILDID=2017-02-02-0235 PRETTYNAME="Container Linux by CoreOS 1235.9.0 (Ladybug)" ANSICOLOR="38;5;75" HOMEURL="" BUGREPORTURL="" Kernel (e.g. ): Linux node-3-vm 4.7.3-coreos-r2 SMP Thu Feb 2 02:26:10 UTC 2017 x8664 Intel(R) Xeon(R) CPU E5-2673 v3 @ 2.40GHz GenuineIntel GNU/Linux Install tools: own Others: What happened: Created new deployment by altering the image tag of the deployment descriptor and using kubectl apply -f "". Noticed that one container of the previous deployment stayed there. What you expected to happen: All containers from the previous deployment should have been terminated. How to reproduce it (as minimally and precisely as possible): I cannot reliably trigger this, I do several deployments per hour and sometimes this problem manifests itself. Anything else we need to know: Deployment descriptor:
Can you also post the output of kubectl get deployment tviftttapp -o yaml kubectl get rs (dangling rs) -o yaml When this manifests again?
Here are the current deployment, the dangling RS, and the "good" RS. Deployment: Dangling RS: Current RS:
It seems that status.availableReplicas for the new replica set is never correctly updated to 2. Short term if you stop using minReadySeconds (minReadySeconds=2 doesn't offer you much anyway), this problem should stop. Long term, we should fix the status update. cc:
can one of you guys paste logs from the controller manager when this issue occurs? Log level should be ideally set to 2.

I managed to reproduce it:
Will close this one once is cherry-picked into 1.5
4 is out and contains a fix that should force the Deployment to be resynced eventually and have availableReplicas updated. Not optimal yet but eventually consistent. I have queued up some follow-up work to refine the resync.
Does anyone know when version 1.5.4 OR 1.6 will be available using Redhat package downloads (in RPM form) ? I only see version 1.5.2 being the latest there. I have the same problem that is being discussed in this thread and want to upgrade so that my K8S cluster can support Deployments.


Question: }, }, }, <ins> { name:      "v1 request, updated endpoints with topology node names + other topology fields", v1Request: true, originalEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1", "other": "value"}, }, { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1", "foo": "bar"}, }, }, }, newEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1a"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1", "other": "value"}, }, { Hostname:           utilpointer.StringPtr("hostname-1b"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1", "foo": "bar"}, }, }, }, expectedEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname: utilpointer.StringPtr("hostname-1a"), NodeName: utilpointer.StringPtr("node-1"), }, { Hostname: utilpointer.StringPtr("hostname-1b"), NodeName: utilpointer.StringPtr("node-1"), }, }, }, }, { name:      "v1 request, updated endpoints with topology node names", v1Request: true, originalEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, }, }, newEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1a"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-1b"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, }, }, expectedEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname: utilpointer.StringPtr("hostname-1a"), NodeName: utilpointer.StringPtr("node-1"), }, { Hostname: utilpointer.StringPtr("hostname-1b"), NodeName: utilpointer.StringPtr("node-1"), }, }, }, }, { name:      "v1 request, updated endpoints with topology node names swapped", v1Request: true, originalEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-2"}, }, }, }, newEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1a"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-2"}, }, { Hostname:           utilpointer.StringPtr("hostname-1b"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, }, }, expectedEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname: utilpointer.StringPtr("hostname-1a"), NodeName: utilpointer.StringPtr("node-2"), }, { Hostname: utilpointer.StringPtr("hostname-1b"), NodeName: utilpointer.StringPtr("node-1"), }, }, }, }, { name:      "v1 request, updated endpoints with new topology node name", v1Request: true, originalEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-2"}, }, }, }, newEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname: utilpointer.StringPtr("hostname-1a"), // Invalid node name because it did not exist in previous version of EndpointSlice DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-3"}, }, { Hostname:           utilpointer.StringPtr("hostname-1b"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, }, }, expectedEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname: utilpointer.StringPtr("hostname-1a"), }, { Hostname: utilpointer.StringPtr("hostname-1b"), NodeName: utilpointer.StringPtr("node-1"), }, }, }, }, { name:      "v1 request, updated endpoints with topology node names + 1 new node name", v1Request: true, originalEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, }, }, newEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1a"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-1b"), NodeName:           utilpointer.StringPtr("node-2"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, }, }, expectedEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname: utilpointer.StringPtr("hostname-1a"), NodeName: utilpointer.StringPtr("node-1"), }, { Hostname: utilpointer.StringPtr("hostname-1b"), NodeName: utilpointer.StringPtr("node-2"), }, }, }, }, { name:      "v1 request, updated endpoints with topology node names + new node names", v1Request: true, originalEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, }, }, newEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1a"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, NodeName:           utilpointer.StringPtr("node-1"), }, { Hostname:           utilpointer.StringPtr("hostname-1b"), NodeName:           utilpointer.StringPtr("node-2"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "node-1"}, }, }, }, expectedEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname: utilpointer.StringPtr("hostname-1a"), NodeName: utilpointer.StringPtr("node-1"), }, { Hostname: utilpointer.StringPtr("hostname-1b"), NodeName: utilpointer.StringPtr("node-2"), }, }, }, }, { name:      "v1 request, invalid node name label", v1Request: true, originalEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "valid-node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-2"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "invalid node-2"}, }, { Hostname:           utilpointer.StringPtr("hostname-3"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "valid-node-3"}, }, { Hostname:           utilpointer.StringPtr("hostname-4"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "invalid node-4"}, }, }, }, newEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname:           utilpointer.StringPtr("hostname-1"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "valid-node-1"}, }, { Hostname:           utilpointer.StringPtr("hostname-2"), DeprecatedTopology: map[string]string{corev1.LabelHostname: "invalid node-2"}, }, { Hostname: utilpointer.StringPtr("hostname-3"), NodeName: utilpointer.StringPtr("node-3"), }, { Hostname: utilpointer.StringPtr("hostname-4"), NodeName: utilpointer.StringPtr("node-4"), }, }, }, expectedEPS: &discovery.EndpointSlice{ Endpoints: []discovery.Endpoint{ { Hostname: utilpointer.StringPtr("hostname-1"), NodeName: utilpointer.StringPtr("valid-node-1"), }, { Hostname: utilpointer.StringPtr("hostname-2"), }, { Hostname: utilpointer.StringPtr("hostname-3"), NodeName: utilpointer.StringPtr("node-3"), }, { Hostname: utilpointer.StringPtr("hostname-4"), NodeName: utilpointer.StringPtr("node-4"), }, }, }, }, </ins> } for _, tc := range testcases {

On the first update of an EndpointSlice using the v1 API, we drop deprecatedTopology with EndpointSlice strategy. Although we only do this if at least 1 endpoint has been modified, there is no guarantee that node name has been populated for all endpoints within the EndpointSlice. This is most noticeable if the first update using the v1 API is only adding endpoint(s). In that case the EndpointSlice controller does not update EndpointSlices and instead just adds the new endpoint(s). That will cause node names to be dropped from the deprecated topology field without having the newer node name field populated. Node name to be retained in deprecated topology until the new field was populated. an EndpointSlice with the v1beta1 API. an endpoint in that EndpointSlice: No response This is most notable after upgrading a 1.20 cluster to 1.21, but can technically occur in any 1.21 or newer cluster. Any Any Any N/A N/A
I'm working on a fix for this, hoping to have it out later today. /triage accepted /sig network /cc
I think this is better characterized as a controller bug, but is the safest mitigation I can think of in the short term, since we don't want to rewrite 100% of endpointslices on startup. Can you open a separate issue to work through the controller reconciliation so that v1 clients can depend on the nodeName field more reliably?