Question: switch { case t.ExitCode == 1: // expected <del> case t.ExitCode == 128 && t.Reason == "ContainerCannotRun" && reBug88766.MatchString(t.Message): </del> <ins> case t.ExitCode == 128 && (t.Reason == "StartError" || t.Reason == "ContainerCannotRun") && reBug88766.MatchString(t.Message): </ins> // pod volume teardown races with container start in CRI, which reports a failure framework.Logf("pod %s on node %s failed with the symptoms of https://github.com/kubernetes/kubernetes/issues/88766") default:

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: Triage board shows flakes here spiked dramatically on 3/4 /sig node /priority critical-urgent /milestone v1.18
are you able to assist if this is related to recent topology work?
sure, I'll take a look
is this the test you just in ?
As mentioned above, this is a new test that was just , and the test sometimes fails. I don't see that any of the recent topology manager changes would cause these failures. I'll be happy to help out and dig more if or others think otherwise.
This is me, the bugs are untriaged containerd / Kubelet failures with low occurrence rates that this test now exposes. Investigating.
still flaking in non-dockershim environments: pod pod-submit-status-0-14 on node kind-worker2 container unexpected exit code 128: start=1970-01-01 00:00:00 +0000 UTC end=2020-03-06 16:16:54 +0000 UTC reason=StartError message=failed to create containerd task: OCI runtime create failed: starting container process caused " container init caused " mounting "/var/lib/kubelet/pods/-819e-4840-8369-" to rootfs "" at "" caused "stat /var/lib/kubelet/pods/-819e-4840-8369- no such file or directory""": unknown `
Different reason this time. Will continue to enumerate failure modes.

If a pod is created and then deleted quickly, volume teardown and the container runtime can race in a way that causes the container runtime to fail to start the container and then report an error. This is because the teardown and container start happen in parallel and cannot synchronize (container start is a call to the container runtime), so by the time the container runtime begins standing up the container the volume is unmounted. makes this flake reproducible because it explicitly tests create-delete-measure and does not exclude this specific flake. We can exclude the flake while this bug is worked. From Found in
/sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /assign
Reopened this issue. <detailsIn response to : /reopen /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /triage accepted
Recent occurrences can be found here
I was hoping this would be but I still see some instances of it in gubernator.
Issues go stale after 90d of inactivity. Mark the issue as fresh with .

Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: func NewForConfig(c *rest.Config) (*Clientset, error) { configShallowCopy := *c <ins> if configShallowCopy.UserAgent == "" { configShallowCopy.UserAgent = rest.DefaultKubernetesUserAgent() } </ins> // share the transport between all clients httpClient, err := rest.HTTPClientFor(&configShallowCopy) if err != nil {

It looks like kubelet started using generic user agent like "Go-http-client/2.0" and makes it harder to attribute the request to kubelet. In the past we have used "kubelet" user-agent which was providing some value. Sample logs: Kubelet's requests are using customised user-agent Run kubelet Check kube-apiserver's logs: No response master tested on gce <details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
The default user agent name is generated like below. I am using v1.23.4 and the user agent can show kubelet as the user agent.(not on gce)
/cc We should identify those and set the user-agent correctly , is right, it is very useful field ... but I see ~kubectl~ kubelet user-agents in the log linked
Interesting... in I see both kubelet user agent and "Go-http-client/2.0": Moreover, it looks like "kubelet" user agent is used only for events-related calls and other (like leases) are using "Go-http-client/2.0".
interesting , the user agent is to the request by wrapping the roundtripper, that is when the client is generated the clientset creation is here Speculating, I'm wondering if the updateTransport is mutating the transport and not adding the corresponding wrapper in ?
/assign I think that this was my fault, let me check it in more detail
I think that this was the problem


Question: return nil } <del> // IsCoreDNSConfigMapMigrationRequired checks if a migration of the CoreDNS ConfigMap is required. func IsCoreDNSConfigMapMigrationRequired(corefile string) bool { </del> <ins> // isCoreDNSConfigMapMigrationRequired checks if a migration of the CoreDNS ConfigMap is required. func isCoreDNSConfigMapMigrationRequired(corefile, currentInstalledCoreDNSVersion string) (bool, error) { var isMigrationRequired bool </ins> if corefile == "" || migration.Default("", corefile) { <del> return false </del> <ins> return isMigrationRequired, nil } deprecated, err := migration.Deprecated(currentInstalledCoreDNSVersion, kubeadmconstants.CoreDNSVersion, corefile) if err != nil { return isMigrationRequired, errors.Wrap(err, "unable to get list of changes to the configuration.") </ins> } <del> return true } func migrateCoreDNSCorefile(client clientset.Interface, cm *v1.ConfigMap, corefile, currentInstalledCoreDNSVersion string) error { // Take a copy of the Corefile data as `Corefile-backup` and update the ConfigMap // Also point the CoreDNS deployment to the `Corefile-backup` data. </del> <ins> // Check if there are any plugins/options which needs to be removed or is a new default for _, dep := range deprecated { if dep.Severity == "removed" || dep.Severity == "newDefault" { isMigrationRequired = true } } </ins> <del> if _, err := client.CoreV1().ConfigMaps(cm.ObjectMeta.Namespace).Update(context.TODO(), &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{ Name:      kubeadmconstants.CoreDNSConfigMap, Namespace: metav1.NamespaceSystem, }, Data: map[string]string{ "Corefile":        corefile, "Corefile-backup": corefile, </del> <ins> return isMigrationRequired, nil } func isCoreDNSVersionSupported(client clientset.Interface) (bool, error) { isValidVersion := true coreDNSPodList, err := client.CoreV1().Pods(metav1.NamespaceSystem).List( context.TODO(), metav1.ListOptions{ LabelSelector: "k8s-app=kube-dns", </ins> }, <del> }, metav1.UpdateOptions{}); err != nil { return errors.Wrap(err, "unable to update the CoreDNS ConfigMap with backup Corefile") </del> <ins> ) if err != nil { return false, errors.Wrap(err, "unable to list CoreDNS pods") </ins> } <del> if err := patchCoreDNSDeployment(client, "Corefile-backup"); err != nil { return err </del> <ins> for _, pod := range coreDNSPodList.Items { imageID := strings.Split(pod.Status.ContainerStatuses[0].ImageID, ":") if !migration.Released(imageID[2]) { isValidVersion = false } </ins> } <ins> return isValidVersion, nil } func migrateCoreDNSCorefile(client clientset.Interface, cm *v1.ConfigMap, corefile, currentInstalledCoreDNSVersion string) error { </ins> // Since the current configuration present is not the default version, try and migrate it. updatedCorefile, err := migration.Migrate(currentInstalledCoreDNSVersion, kubeadmconstants.CoreDNSVersion, corefile, false) if err != nil { return errors.Wrap(err, "unable to migrate CoreDNS ConfigMap") } <ins> // Take a copy of the existing Corefile data as `Corefile-backup` and update the ConfigMap </ins> if _, err := client.CoreV1().ConfigMaps(cm.ObjectMeta.Namespace).Update(context.TODO(), &v1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{ Name:      kubeadmconstants.CoreDNSConfigMap,

What happened: Coredns failed to start in 1.17 deployment when it tried to retain a backed up Corefile with error message: Caddyfile via flag: open /etc/coredns/Corefile: no such file or directory What you expected to happen: Coredns starts normally with the backed-up Corefile How to reproduce it (as minimally and precisely as possible): Deploy a k8s 1.17 cluster and trigger coredns Corefile migration and make it fail. In our case, this was triggered in a three master -setup with 'old' method of running 'kubeadm init' for each master. Also the core-dns docker image used was tagged with '1.6.7-1', apparently causing the corefile migration version check to fail Anything else we need to know?: This seems to be caused by Both 'key' and 'path' are set to 'Corefile-backup' when this function is called. Should the 'path' always stay as 'Corefile' because that is where the container expects to find the config file? Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-16T17:51:57Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-16T17:47:13Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Custom deployment in Openstack environment OS (e.g: ): NAME="SLES" VERSION="15-SP1" VERSIONID="15.1" PRETTYNAME="SUSE Linux Enterprise Server 15 SP1" Kernel (e.g. ): 12.14-197.29-default Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
/assign
/area kubeadm


Question: } } <del> func TestValidateToleratingBadLabels(t *testing.T) { invalidSelector := getValidLabelSelector() invalidSelector.MatchExpressions = []metav1.LabelSelectorRequirement{{Key: "key", Operator: metav1.LabelSelectorOpNotIn, Values: []string{"bad value"}}} validPodTemplateSpec := getValidPodTemplateSpecForSelector(getValidLabelSelector()) job := &batch.Job{ ObjectMeta: getValidObjectMeta(0), Spec: batch.JobSpec{ Selector:       invalidSelector, ManualSelector: pointer.BoolPtr(true), Template:       validPodTemplateSpec, }, } job.ResourceVersion = "1" oldObj := job.DeepCopy() newObj := job.DeepCopy() context := genericapirequest.NewContext() errorList := Strategy.ValidateUpdate(context, newObj, oldObj) if len(errorList) > 0 { t.Errorf("Unexpected error list with no-op update of bad object: %v", errorList) } } func TestJobStrategyValidateUpdate(t *testing.T) { </del> <ins> func TestJobStrategy_ValidateUpdate(t *testing.T) { </ins> ctx := genericapirequest.NewDefaultContext() validSelector := &metav1.LabelSelector{ MatchLabels: map[string]string{"a": "b"},

The gets very long testing mulitple functions in such as , and . These checks should be performed in dedicated Test functions using table for different scenarios. Target functions and are being as part of The remaining checks should be migrated to these functions. Also, a test function should be to migrate the remaining tests.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Any progress on this? I'm working on and I noticed that has relatively low coverage so I was going to work on getting the coverage up.
I created that is somewhat related to this.
no progess. go ahead please
/assign


Question: --env "KUBE_FASTBUILD=${KUBE_FASTBUILD:-false}" --env "KUBE_BUILDER_OS=${OSTYPE:-notdetected}" --env "KUBE_BUILD_PPC64LE=${KUBE_BUILD_PPC64LE}"  # TODO(IBM): remove <ins> --env "KUBE_VERBOSE=${KUBE_VERBOSE}" </ins> ) # If we have stdin we can run interactive.  This allows things like 'shell.sh'

Similar to this is to test the first contributor experience fails on ubuntu 16.04, docker version 1.12.3:
Got the same error on Linux ubuntu 4.2.0-27-generic ~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x8664 x8664 x86_64 GNU/Linux Docker Client version: 1.6.2 Client API version: 1.18 Go version (client): go1.2.1 Git commit (client): OS/Arch (client): linux/amd64 Server version: 1.6.2 Server API version: 1.18 Go version (server): go1.2.1 Git commit (server): OS/Arch (server): linux/amd64
What git sha of were you building from?
I have cloned the head. I think the reason might be missing package: gccgo-4.8-aarch64-linux-gnu I'm compiling right now and update on result.
were you running or ? runs inside docker, so the gcc error is confusing.
I run: sudo make release in Ubuntu VM running in VMWare workstation if that matters. Are you in kubernetes-users slack channel?
does fail in the same way?
Do you need to run , instead of just ? Can you try running the following:
well, if I don't do sudo docker does not work :) I need to run as root. I'll try the commands as soon as quick-release ends.
ah, gotcha. you can probably fix the docker access issue by following in the docker manual. that's probably not why it's failing, though.
make release still does not work :(
can you please run and then paste the results? ( turns up the debugging verbosity.)
for me: works (using your PR to fixes the gobindata dep) -local Go build- works (docker build) works if you have Docker running. does not work: on OSX did not seem to give more verbosity.
V=4 make release did not help :(
oops, that was not the correct environment variable, sorry. I meant .
here we go:
Aha, I think I figured it out. Sorry for making you rebuild so many times; the issue still isn't super obvious from the logs.

Basically, the bug only occurs if you don't have enough memory to do the docker builds in parallel; if that happens, you fall into this . On the parallel side of the branch, we set the platform env vars within a subshell, so each of the cross-compilation builds won't affect each other. On the non-parallel side of the branch, we don't use a subshell. Inside , there's no overrides done for . As a result, we end up incorrectly reusing the value for , , which doesn't recognize the flag being applied for the darwin build. The fix should be easy; I'll get that sent out ASAP. we probably want this fix for 1.5, since if you fall back to non-parallel crossbuilds we're doing the wrong thing.
Ack. Will keep an eye out for the fix. Please mark it as cherrypick-candidate
Thank you guys!
I think it is almost there. Got much further in the build (4 GB of RAM). Towards the end there is a failed test that makes the build fail: This is probably a different bug, as your PR got passed the misconfigured compilation.
FWIW, the worked on ubuntu 16.06. It did ran some tests apparently but the build succeeded.
I just tried with and it worked fine
since we have a PR ready and working, can we please switch this to non-release-blocker?


Question: if err != nil || len(externalDNS) == 0 { //TODO: It would be nice to be able to determine the reason for the failure, // but the AWS client masks all failures with the same error description. <del> glog.V(2).Info("Could not determine public DNS from AWS metadata.") </del> <ins> glog.V(4).Info("Could not determine public DNS from AWS metadata.") </ins> } else { addresses = append(addresses, v1.NodeAddress{Type: v1.NodeExternalDNS, Address: externalDNS}) }

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig aws What happened: Every few seconds I get spam in my logs about "Could not determine public DNS from AWS metadata" from every node. This is really very spammy. What you expected to happen: No spam. How to reproduce it (as minimally and precisely as possible): Launch a private topology cluster with kops. Anything else we need to know?: There will never be a public-hostname in the metadata for private topology clusters. This line: Should be glog.V(4) i suspect, like line 1033 above it. Environment: Kubernetes version (use ): 1.7.4 Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Jessie Kernel (e.g. ): 4.4.78-k8s Install tools: kops Others:
fixed in . PTAL. Thanks!
Thanks On a side note the metadata should probably be cached to stop thrashing the API but I don't know enough to be sure.
/cc


Question: legacyregistry.MustRegister(RunningContainerCount) legacyregistry.MustRegister(RunningPodCount) legacyregistry.MustRegister(ManagedEphemeralContainers) <ins> if utilfeature.DefaultFeatureGate.Enabled(features.KubeletPodResources) { legacyregistry.MustRegister(PodResourcesEndpointRequestsTotalCount) if utilfeature.DefaultFeatureGate.Enabled(features.KubeletPodResourcesGetAllocatable) { legacyregistry.MustRegister(PodResourcesEndpointRequestsListCount) legacyregistry.MustRegister(PodResourcesEndpointRequestsGetAllocatableCount) legacyregistry.MustRegister(PodResourcesEndpointErrorsListCount) legacyregistry.MustRegister(PodResourcesEndpointErrorsGetAllocatableCount) } } </ins> legacyregistry.MustRegister(StartedPodsTotal) legacyregistry.MustRegister(StartedPodsErrorsTotal) legacyregistry.MustRegister(StartedContainersTotal)

In commit we podresources-related metrics, to make sure the feature was monitoring-friendly. Because of a implementation bug, the metrics have been , they are updated correctly, but they are never registered, so they don't show up when doing like the podresources related metrics should be reported in the endpoint in the kubelet the kubelet, even the simplistic environment of the tests is sufficient sure to enable the and feature gates (on by default in kube = 1.25 at least) the podresources endpoint issuing and calls. The actual output is not relevant the metrics: The bug was introduced in the original commit <details</details<detailsNot relevant. </details<detailsNot relevant </details<detailsNot relevant </details<detailsNot relevant </details<detailsNot relevant </details
/sig node /assign
/cc /triage-accepted
/triage accepted /cc


Question: .nf File Path passed as a flag on the command line. This file is rechecked every 20 seconds (configurable with a flag). HTTP endpoint HTTP endpoint passed as a parameter on the command line. This endpoint is checked every 20 seconds (also configurable with a flag). <del> etcd server The Kubelet will reach out and do a watch on an etcd server. The etcd path that is watched is /registry/hosts/$(hostname -f). As this is a watch, changes are noticed and acted upon very quickly. </del> <ins> etcd server The Kubelet will reach out and do a watch on an etcd server. The etcd path that is watched is /registry/hosts/$(uname -n). As this is a watch, changes are noticed and acted upon very quickly. </ins> HTTP server The kubelet can also listen for HTTP and respond to a simple API (underspec'd currently) to submit a new manifest. .fi

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: container_memory_working_set_bytes{container="container_a",namespace="namespace_b",pod="pod_b"} 1000 1624396278302 # HELP container_start_time_seconds [STABLE] Start time of the container since unix epoch in seconds # TYPE container_start_time_seconds gauge <del> container_start_time_seconds{container="container_a",namespace="namespace_a",pod="pod_a"} 1.6243962483020916e+09 1624396248302 container_start_time_seconds{container="container_a",namespace="namespace_b",pod="pod_b"} 1.6243956783020916e+09 1624395678302 </del> <ins> container_start_time_seconds{container="container_a",namespace="namespace_a",pod="pod_a"} 1.6243962483020916e+09 container_start_time_seconds{container="container_a",namespace="namespace_b",pod="pod_b"} 1.6243956783020916e+09 </ins> # HELP scrape_error [ALPHA] 1 if there was an error while getting container metrics, 0 otherwise # TYPE scrape_error gauge scrape_error 0

When scrape kubelet resource metrics provided by /metrics/resource, got response like this The metric containerstarttimeseconds's timestamp should not equals to container start time As above No response v1.24.7 Volc Engine No response No response No response No response_
I would like to take this up, after triage/accepted I just wanted to know then what time should be needed to be set there? /assign
the metric should not have a timestamp
If not timestamp, then what value and data type should I assign for the containerstarttime_seconds metric?
check
no need to change the value and data type, just remove the timestamp for the metric datapoint
Sure got it, raising a PR for the same!
If you have already done it, then why the PR is blocked and why I am redoing it again? Just curious to know!
Why do you think it's blocked and why are you redoing it before figuring this out?
/sig node
/triage accepted /priority important-longterm
/assign (since you sent a PR)


Question: for _, pvTopology := range podPVTopologies { v, ok := node.Labels[pvTopology.key] <ins> if !ok { // if we can't match the beta label, try to match pv's beta label with node's ga label v, ok = node.Labels[translateToGALabel(pvTopology.key)] } </ins> if !ok || !pvTopology.values.Has(v) { logger.V(10).Info("Won't schedule pod onto node due to volume (mismatch on label key)", "pod", klog.KObj(pod), "node", klog.KObj(node), "PV", klog.KRef("", pvTopology.pvName), "PVLabelKey", pvTopology.key) return framework.NewStatus(framework.UnschedulableAndUnresolvable, ErrReasonConflict)

In , we discovered that our plan for eventually removing the scheduling plugin when CSI migration is GA is not possible because the plugin also applies to non-migratable volume types. However, if we leave the plugin as is, then even migratable PVs will be broken if we remove beta topology labels from Nodes in the future. This is because the plugin does not have any handling for migrated volumes. I suggest we add CSI migration handling to the VolumeZone plugin. I think we just need to detect if the volume is migrated, and if so, skip the VolumeZone processing. For migrated volumes, we translate into so they end up getting processed by the plugin. In a cluster with csi migration for a specific intree volume type enabled, an intree PV with beta topology labels the beta topology label from a node to schedule a pod using that pv No response <details</details<detailsAny that supports csi migration </details<details</details<details</details<details</details<details</details
/sig storage /triage accepted /priority important-soon For e2e testing, this needs an environment that is configured with csi migration
Hi,I'd like to work on this issue. I previously wrote for plugin, so I'm familiar with it's logic. However I'm not familiar with the whole thing, and I may ask you some questions during the work :( Can I try this one? Here are some questions ahead: I think what you want to do is to schedule a pv with old labels to a node with only new labels. And you think we can translate into in volumezone during (maybe), and update this pv so that in stage, can , so we don't have compare Topology labels in . My question is: Can't we just translate this for all pvs? Why do we need to detect if the volume is migrated? What's the difference between a migrated pv and a non-migratable pv in this scenario?
I think this is also another alternative solution that could also work, but from , it may not scalable to do an automatic migration for all the different APIs and controllers that use the node labels. The main difference with CSI migration, is we have a good idea of all the different components that would be impacted and also have reasonable control over those components.
A few more questions Will the volumes be migrated before we take a pod off the scheduler queue?

If so, how can I detect this. Currently I find a function in , but nothing like . Is this translation done in plugin or before scheduling? I think currently in , we consider topology label and beta label as different labels. That is ,if we have a pv with and a node with , even if they have same values, will let the pod pass filter. If we're going to remove beta labels on node,maybe we should try to consider two labels as the same in ? Here's my understanding: For migrated volumes, we translate Topology labels to pv.NodeAffinity and update the pv, then we skip volume_zone(that is,don't compare labels with nodes). What about non-migratable volumes, I think we should continue to compare with node labels for non-migratable volumes?
For detecting if a PV is migrated, you can use the same method that's used in the VolumeBinding plugin: The only difference is you don't actually need to translate the PV in the VolumeZone plugin since it's already handled by the VolumeBinding plugin. For non-migratable volumes, I think we can continue with the same VolumeZone logic. For now, we'll rely on users to update their PVs with the GA labels.
Thanks ,for your detailed explanation. This is the implementation plan I prepared: In , we check if a pv is migrtable by using ,if so we skip volumezone since will translate this pv and handle it later, otherwise we save the label of this pv in state,and compare it with node's label in .Is that feasible to you?
Yup that sounds good to me!
Thanks! I'll try to work on this one. I'll file a pr a few days later and we can talk about the details in there. In the meantime, I think I previously wrote a bug in plugin, I wonder if you have time to confirm it /assign
This issue is labeled with but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release. You can: Confirm that this issue is still relevant with (org members only) Deprioritize it with or Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted


Question: --name "${KUBE_DATA_CONTAINER_NAME}" --hostname "${HOSTNAME}" "${KUBE_BUILD_IMAGE}" <del> chown -R $(id -u).$(id -g) </del> <ins> chown -R ${USER_ID}.${GROUP_ID} </ins> "${REMOTE_ROOT}" /usr/local/go/pkg/ )

Is this a BUG REPORT or FEATURE REQUEST? (choose one):BUG REPORT Kubernetes version (use ):Client Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2850+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-01-06T09:06:04Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6+", GitVersion:"v1.6.0-alpha.0.2753+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-01-04T08:46:56Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} What happened: How to reproduce it (as minimally and precisely as possible): the owner of file is not right
Do you need to be building as root?
I get an annoying message when building as normal user: So I run as root. There will return a error if the outpt build directory is not owned by root when running as root. I think we could use to fix it.
I hope to get some of your suggestions
did you previously build as ? you might need to before a build as a normal user works again.


Question: "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/latest" "k8s.io/kubernetes/pkg/runtime" <ins> // This installs the legacy v1 API _ "k8s.io/kubernetes/pkg/api/install" </ins> ) // TODO:

I follow to create a nginx example. And I got stuck at the beginning.
I'll send a fix which will supersede .
that would be great! thanks, I just got lost handling other 1.1 stuff and forgot about that pr
Sent , PTAL. Thank you.


Question: "x-kubernetes-patch-strategy": "merge" }, "generateName": { <del> "description": "GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.nnIf this field is specified and the generated name exists, the server will NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header).nnApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#idempotency", </del> <ins> "description": "GenerateName is an optional prefix, used by the server, to generate a unique name ONLY IF the Name field has not been provided. If this field is used, the name returned to the client will be different than the name passed. This value will also be combined with a unique suffix. The provided value has the same validation rules as the Name field, and may be truncated by the length of the suffix required to make the value unique on the server.nnIf this field is specified and the generated name exists, the server will return a 409.nnApplied only if Name is not specified. More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#idempotency", </ins> "type": "string" }, "generation": {

The error code of is 409. Is this an expected behavior? The doc seems to say no: NOT return a 409 - instead, it will either return 201 Created or 500 with Reason ServerTimeout indicating a unique name could not be found in the time allotted, and the client should retry (optionally after the time indicated in the Retry-After header). seems to create which error code is 409. The error code of isn't 409. Or, the doc doesn't say "If this field is specified and the generated name exists, the server will NOT return a 409". Create a large number of Pods with generateName, and cause name conflicts. Just question: if the API is correct in returning 409 for a GenerateName-conflict, how do we determine that error? We want to retry when GenerateName conflict returns in most cases because the retry may solve this conflict. latest master (maybe after ) <details</details<details</details<details</details<details</details<details</details
Sorry if not an appropriate SIG. /sig api-machinery
/assign
It seems related to . If it's expected behavior, the document isn't correct..?
is a mistake. /unassign
Hi do you think this is a good first issue to tackle? If so, I would like to try contributing here.
/assign /triage accepted
Forgive me to ping other people who might know. /cc /cc /cc
I suspect we just failed to update the documentation.
Thanks Okay. I'll create the PR for that. /kind documentation /assign


Question: close(pk.podKillingCh) } <del> // KillPod sends pod killing request to the killer func (pk *podKillerWithChannel) KillPod(pair *kubecontainer.PodPair) { pk.podKillingCh <- pair } </del> // PerformPodKillingWork launches a goroutine to kill a pod received from the channel if // another goroutine isn't already in action. func (pk *podKillerWithChannel) PerformPodKillingWork() { for podPair := range pk.podKillingCh { <del> if pk.checkAndMarkPodPendingTerminationByPod(podPair) { // Pod is already being killed continue } </del> runningPod := podPair.RunningPod apiPod := podPair.APIPod

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Right after calling , returns false even the pod is not killed yet. What you expected to happen: In kubelet, after is complete, the pod should be pending termination or be killed. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Related to Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node /assign


Question: "k8s.io/mount-utils" utilpath "k8s.io/utils/path" <del> libcontaineruserns "github.com/opencontainers/runc/libcontainer/userns" </del> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" "k8s.io/apimachinery/pkg/types"

is no longer compilable on macOS in the master branch. It was compilable with Kubernetes v1.29.0 and v1.30.0-alpha.0. It should be compilable. No response <detailsmaster () (This is just the latest commit, not the commit that introduced the regression) </details<detailsN/A </details<detailsmacOS 14.2.1 (Intel) </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
Regression in:


Question: ClientConfig: kubeconfig, } var clientBuilder controller.ControllerClientBuilder <del> if len(s.ServiceAccountKeyFile) > 0 && s.UseServiceAccountCredentials { </del> <ins> if s.UseServiceAccountCredentials { if len(s.ServiceAccountKeyFile) > 0 { // It's possible another controller process is creating the tokens for us. // If one isn't, we'll timeout and exit when our client builder is unable to create the tokens. glog.Warningf("--use-service-account-credentials was specified without providing a --service-account-private-key-file") } </ins> clientBuilder = controller.SAControllerClientBuilder{ ClientConfig:         restclient.AnonymousClientConfig(kubeconfig), CoreClient:           kubeClient.CoreV1(),

/kind bug What happened: I'm unable to set up a k8s cluster with RBAC due authorization errors (guessing) Nor set up kube-dns How to reproduce it (as minimally and precisely as possible): Run each component with these options: Anything else we need to know?: PRs and were patched against and kubelet + kube-controller-manager are being executed with those patches in place. Since those patches are IPv6 related, they should not affect the errors I'm seeing. Environment: Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig auth
you are telling the controller-manager to run its control loops using service account credentials (the flag), but you are not giving the controller manager a so it can actually create those credentials. You would need to specify that, and also give the corresponding public key to the apiserver (with ) so the credentials are recognized.
Opened to make it clearer when the controller manager is in this situation (it would log a warning and fail to start, rather than continue incorrectly)


Question: <ins> #!/bin/bash # Copyright 2014 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Creates resources from the example, assumed to be run from Kubernetes repo root export OPENSHIFT_EXAMPLE=$(pwd)/examples/openshift-origin export OPENSHIFT_CONFIG=${OPENSHIFT_EXAMPLE}/config mkdir ${OPENSHIFT_CONFIG} cluster/kubectl.sh config view --output=yaml --flatten=true --minify=true > ${OPENSHIFT_CONFIG}/kubeconfig cluster/kubectl.sh create -f $OPENSHIFT_EXAMPLE/openshift-service.yaml sleep 30 export PUBLIC_IP=$(cluster/kubectl.sh get services openshift --template="{{ index .spec.publicIPs 0 }}") echo $PUBLIC_IP export PORTAL_IP=$(cluster/kubectl.sh get services openshift --template="{{ .spec.portalIP }}") echo $PORTAL_IP docker run --privileged -v ${OPENSHIFT_CONFIG}:/config openshift/origin start master --write-config=/config --kubeconfig=/config/kubeconfig --master=https://localhost:8443 --public-master=https://${PUBLIC_IP}:8443 sudo -E chown ${USER} -R ${OPENSHIFT_CONFIG} docker run -i -t --privileged -e="OPENSHIFTCONFIG=/config/admin.kubeconfig" -v ${OPENSHIFT_CONFIG}:/config openshift/origin ex bundle-secret openshift-config -f /config &> ${OPENSHIFT_EXAMPLE}/secret.json cluster/kubectl.sh create -f ${OPENSHIFT_EXAMPLE}/secret.json cluster/kubectl.sh create -f ${OPENSHIFT_EXAMPLE}/openshift-controller.yaml cluster/kubectl.sh get pods | grep openshift </ins>  No newline at end of file

This PR was prematurely merged: Need to update the PR based on the actual steps that were needed to get what was demonstrated at the public Kubernetes meet-up last week.
Any chance you can get rid of the hostPort from the example?
- depending on the Kubernetes provider, yes. For Vagrant based provider, I think I still need the hostPort because I do not have routes. For the GCE provider, I should be able to get rid of it because I have routes. In both providers, there were a number of setup steps that did not make the merged PR. For those that are tracking this issue (which I know folks have pinged me on separately) needs to be configured to allow privelaged containers needs to be configured to allow pushing to an insecure registry (this is point in time) Ideally, I could use a ReplicationController in the example to load balance OpenShift, but until we can provide a pattern to run etcd as a pod, I am unable to do this and need to use the embedded etcd we end up running.
Note: I will resolve this issue in 2 phases. Vagrant (which was demonstrated) to document using GCE (which has some additional steps on firewall rules)
Turns out OpenShift Origin is too secure now ;-) This is blocked until OpenShift Origin allows you to mint certs on the client, and then supply the certs on startup as a secret to the pod. This feature is coming soon. /cc
I now have all the required pieces to get this to work. I hope to get a PR submitted to fix this issue by end of this week.


Question: // So this function needs to be called after the extra envs are applied. func updateImageWhiteList() { // Union NodePrePullImageList and CommonImageWhiteList into the framework image pre-pull list. <del> framework.ImageWhiteList = NodePrePullImageList.Union(commontest.CommonImageWhiteList) </del> <ins> framework.ImageWhiteList = NodePrePullImageList.Union(commontest.PrePulledImages) </ins> // Images from extra envs framework.ImageWhiteList.Insert(getNodeProblemDetectorImage()) framework.ImageWhiteList.Insert(getSRIOVDevicePluginImage())

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We should come up with a better name for : Why is this needed: From /sig testing /priority important-soon /cleanup /help /good-first-issue
/assign
/lifecycle active /remove-help
what happened with ?
I redid it here There was a bit of confusion I'll fix it up.
/wg naming


Question: # to readonly. # The configured vars will only contain platforms allowed by the # KUBE_SUPPORTED* vars at the top of this file. <del> declare -a -g KUBE_SERVER_PLATFORMS declare -a -g KUBE_CLIENT_PLATFORMS declare -a -g KUBE_NODE_PLATFORMS declare -a -g KUBE_TEST_PLATFORMS </del> <ins> declare -a KUBE_SERVER_PLATFORMS declare -a KUBE_CLIENT_PLATFORMS declare -a KUBE_NODE_PLATFORMS declare -a KUBE_TEST_PLATFORMS </ins> kube::golang::setup_platforms() { if [[ -n "${KUBE_BUILD_PLATFORMS:-}" ]]; then # KUBE_BUILD_PLATFORMS needs to be read into an array before the next

Commits and on master via PRs and have broken MacOS 'make' again: where "MacOS" is a non-homebrew extended MacOS with old old bash. This is also reported on other older platforms like CentOS. Master needs fixed with a new commit, then the above PRs and and the new fix need cherry picked to 1.14 as 1.14.2 is currently not buildable as is cherry picked there. :bash_fire"
/sig release /priority critical-urgent
/cc is looking at this


Question: }, fakeCloud: &fake_cloud.FakeCloud{ Machines: []string{"node0", "node1"}, <ins> ExtID: map[string]string{ "node0": "ext-node0", "node1": "ext-node1", }, </ins> }, matchRE:              ".*", expectedRequestCount: 2, // List + Create <del> expectedCreated:      []string{"node1"}, </del> <ins> expectedNameCreated:  []string{"node1"}, expectedExtIDCreated: []string{"ext-node1"}, </ins> expectedDeleted:      []string{}, }, {

I see the following in the kube-controller-manager log: - This is happening to me on HEAD and is the reason why pod status is always showing unknown even when they are scheduled and running.
This is with the vagrant cloud provider. It looks like the delta between the new and old minion is the Node.Spec.ExternalID is being set on the incoming minion but is not set in the originally created minion.
I am not sure how this is not affecting other cloud providers. It looks to me that NodeController populated the node ExternalID here: It looks like the Kubelet provided value is missing the ExternalID
Actually, I am now confused on what the NodeController may or may not be doing here that is causing the status to not get updated.
It looks like the client is passing an ExternalID on the incoming request, but the original value that is in etcd does not have an ExternalID set. Not sure what changed that would have caused this issue to appear.
I think this causes problems: The memory is re-used in a for-loop by Go so referring to a pointer value in a map here will cause problems I think.
Ok, the referenced line is the problem. Sending a PR.
Well, I go back on that now. A simple way to reproduce this is to delete all the minions in your cluster, and verify the node controller is able to add them back. For me, this is causing the node controller to add them back in an unknown status, and then all updates to the node fail. The update fails because the node controller is not setting an ExternalID on the original object.
I am looking into this now. cc/ too
Related issue opened while debugging this:
Debugged through here: I can via manual fmt see that the object has an ExternalID set prior to the call Create. When I look at etcd, I end up with this persisted: Note: it is missing an ExternalID somehow ;-)
Ok, I think I see the issue Nodes returned from here: Do not have an ExternalID Which can make Nodes created from here: End up without an External ID. Having two different nodeClient.Create() paths in this code is confusing. Also we are doing bad things in maps holding pointers to memory allocated in for-loop. We need to fix that ;-)
Yeah, this is bug we should fix. Thanks for looking into this.

If you already have a potential fix, please feel free to grab the issue from me.
I think I have a PR that should fix. Testing that now. Will cc you on the PR.
Thanks for fixing - it did end up affecting AWS as well (AWS isn't in a fully working state just yet, particularly with node discovery).


Question: // set the next loop up options.Continue = m.GetContinue() <del> // Clear the ResourceVersion on the subsequent List calls to avoid the </del> <ins> // Clear the ResourceVersion(Match) on the subsequent List calls to avoid the </ins> // `specifying resource version is not allowed when using continue` error. // See https://github.com/kubernetes/kubernetes/issues/85221#issuecomment-553748143. options.ResourceVersion = "" <ins> options.ResourceVersionMatch = "" </ins> // At this point, result is already paginated. paginatedResult = true }

We recently started using the list pager from client-go as part of Calico, and have seen reports of the following error showing up: No error, just beautiful paged list calls. Create 1,000 resources of a particular type, such as a NetworkPolicy. (Need 500 to trigger paging) Do a of all those resources with set. Originally filed here (but maybe that repo is not monitored): Reference issue in Calico: projectcalico/calico (comment) Reference Calico list pager code: We think this is the required fix: <detailsclient-go version (but bug seems to be in master too). </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/triage accepted


Question: for i := range peerPkgs { peerPkgs[i] = genutil.Vendorless(peerPkgs[i]) } <ins> for i := range extraDirs { extraDirs[i] = genutil.Vendorless(extraDirs[i]) } </ins> // Make sure our peer-packages are added and fully parsed. <del> for _, pp := range peerPkgs { </del> <ins> for _, pp := range append(peerPkgs, extraDirs...) { </ins> context.AddDir(pp) p := context.Universe[pp] if nil == p {

In the codegenerator we have: So basically we process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator. This is problematic, when regenerating conversions, we simply pass to input directories only those that were changed in the meantime: We should fix that - probably by passing all CONVERSIONDIRS as inputs to conversion-gen /kind-bug
I think this caused Not sure if was related yet.
It's the other way around - this is the main bug and root cause and the other that you linked is a consequence.
Also - FYI
I actually already started looking into it - so let me assign this.
is out for review
/triage accepted


Question: "k8s.io/kubernetes/pkg/kubelet/cm" evictionapi "k8s.io/kubernetes/pkg/kubelet/eviction/api" "k8s.io/kubernetes/pkg/kubelet/lifecycle" <ins> "k8s.io/kubernetes/pkg/kubelet/metrics" </ins> kubepod "k8s.io/kubernetes/pkg/kubelet/pod" "k8s.io/kubernetes/pkg/kubelet/qos" "k8s.io/kubernetes/pkg/kubelet/server/stats"

We should keep track of how stale stats used for evictions are. This will help track improvements in eviction latency, and help uncover regressions. cc If someone could assign this to me, I would appreciate that.
closing via


Question: func ParseToRequirements(selector string, opts ...field.PathOption) ([]Requirement, error) { return parse(selector, field.ToPath(opts...)) } <ins> // ValidatedSetSelector wraps a Set, allowing it to implement the Selector interface. Unlike // Set.AsSelectorPreValidated (which copies the input Set), this type simply wraps the underlying // Set. As a result, it is substantially more efficient. A nil and empty Sets are considered // equivalent to Everything(). // // Callers MUST ensure the underlying Set is not mutated, and that it is already validated. If these // constraints are not met, Set.AsValidatedSelector should be preferred // // None of the Selector methods mutate the underlying Set, but Add() and Requirements() convert to // the less optimized version. type ValidatedSetSelector Set func (s ValidatedSetSelector) Matches(labels Labels) bool { for k, v := range s { if !labels.Has(k) || v != labels.Get(k) { return false } } return true } func (s ValidatedSetSelector) Empty() bool { return len(s) == 0 } func (s ValidatedSetSelector) String() string { keys := make([]string, 0, len(s)) for k := range s { keys = append(keys, k) } // Ensure deterministic output sort.Strings(keys) b := strings.Builder{} for i, key := range keys { v := s[key] b.Grow(len(key) + 2 + len(v)) if i != 0 { b.WriteString(",") } b.WriteString(key) b.WriteString("=") b.WriteString(v) } return b.String() } func (s ValidatedSetSelector) Add(r ...Requirement) Selector { return s.toFullSelector().Add(r...) } func (s ValidatedSetSelector) Requirements() (requirements Requirements, selectable bool) { return s.toFullSelector().Requirements() } func (s ValidatedSetSelector) DeepCopySelector() Selector { res := make(ValidatedSetSelector, len(s)) for k, v := range s { res[k] = v } return res } func (s ValidatedSetSelector) RequiresExactMatch(label string) (value string, found bool) { v, f := s[label] return v, f } func (s ValidatedSetSelector) toFullSelector() Selector { return SelectorFromValidatedSet(Set(s)) } var _ Selector = ValidatedSetSelector{} </ins>

A common pattern in Kubernetes (both internally and in third party usage) is something like: (this is from the ). This is commonly called at a fairly high rate; in our case we would actually end up seeing this account for significant portions of CPU/memory to compute the and run on it. In our controller, as an optimization, we replaced usage of and just used our own simpler label matching code, leading to a 70% speed up in our function (which does more than just match) and 96% allocation size improvement; see I believe similar could be done in Kubernetes directly by having a specialized Selector implementation for simple labels. This would transparently given substantial performance benefits to anyone using the library
/sig api-machinery
is the change here a new kind of selector matching or a tweak to something existing? It's hard to tell if this is proposing a new design to partition types of selectors that needs a kep versus a perf tweak to existing code that is straight-forward.
The proposal is an internal performance improvement to make a specialized implementation of the selector matching internally. So two concrete implementations for 1 shared interface. is a PoC of this. I think it can be extended (by me, or others if someone wants to) to complete it
We discussed this one during our bug triage. There seemed to be the feeling that this is large enough to need a KEP? Would you be up for that? Additionally you could bring it up in our SIG meeting to socialize / discuss it more if you wanted.
The patch shared in helps to explain, it's a lot smaller than the description made it sound. This is a special case for equality-only selectors. Probably still worth bringing up at a sig-meeting, but if if the patch is a good representation, it is more bounded than expected from the issue description.
/triage accepted


Question: } } <del> type UnstructuredCreator struct{} </del> <ins> type unstructuredCreator struct{} </ins> <del> func (UnstructuredCreator) New(kind schema.GroupVersionKind) (runtime.Object, error) { </del> <ins> func (c unstructuredCreator) New(kind schema.GroupVersionKind) (runtime.Object, error) { </ins> ret := &unstructured.Unstructured{} ret.SetGroupVersionKind(kind) return ret, nil

BUG REPORT Kubernetes version (use ): v1.6.2 Environment: Cloud provider or hardware configuration: GKE What happened: Create TPR "MyResource" Create namespace "soak-test" Create a "MyResource" "a" under "soak-test":Delete namespace "soak-test"wait namespace to be deleted MyResource "a" still exists after ns is deleted.
/area third-party-resource
/sig cli
I'd like to take a look at the problem if nobody else is working on this
Root Cause of this is the fact that Namespace on ThirdPartyResources is getting ignored (both passed as command --namespace, or the one present in yaml) ThirdPartyResources always end up in default namespace (which is causing confusion reported in bug)
How long has that bug existed? might related to your other findings. We know that bug doesn't exist in CRDs (we have tests specifically checking namespace handling), so I'd suggest converting to those as soon as possible.
Isn't CRD available in 1.7+ ?
Yes. If the bug has existed for multiple releases (not a regression), I don't think this is a blocker for 1.7. It could still be patched, just not a blocker for the current release.
Migration to CRD still is not automated and will still take some time. Have you considered that?
Yes, but if the behavior is not a regression, then this does not block the release. The fact that we have a clear path forward to a working resource is gravy.
This bug existed in 1.4 (), got fixed in 1.5, but appears again in 1.6 .
Ok. Not a blocker for 1.7, but if the patch is reasonable as a fix we can merge it into the service stream. For 1.7, I'd still recommend moving to CRD as quickly as possible.
I was able to reproduce this with CRD on master HEAD. What level of tests are there? Anything e2e?
most of them and specifically a storage one. You're saying we have the from above?
No, I don't see any evidence that they always end up in the default namespace (neither for TPR nor CRD). I'm not sure what that's referring to.

What I was able to reproduce is the bug described at the top of this issue: Create item in namespace, delete namespace, item remains. I observed this for TPR on and , and for CRD on .
The problem seems to be that is misspelled as in this line: If I fix that, namespace cleanup works for TPR. However, for CRD, now namespace cleanup gets stuck and controller manager logs this: Edit: To clarify, I "fixed" the check in the linked code by removing the entire highlighted block. It seems like the singular is not the only problem. It was also using the internal version of when was expected.
Regarding namespace cleanup getting stuck for CRD after the "fix", the error on the apiserver side is: According to the stack trace, the error comes from here:
yes, not only the is misspelled, but also the mismatch of the version. this is why in my original patch, I tried to ignore the version of .
I'd like to keep this open until we have a regression test since this is at least the third time we've fixed this. I'll add a test as part of making sure this fix works for CRD.


Question: glog.V(1).Infof("Pulling image %s without credentials", image) } <del> return p.client.PullImage(opts, creds) </del> <ins> err := p.client.PullImage(opts, creds) // If there was no error, or we had credentials, just return the error. if err == nil || ok { return err } // Image spec: [<registry>/]<repository>/<image>[:<version] so we count '/' explicitRegistry := (strings.Count(image, "/") == 2) glog.Errorf("Foo: %s", explicitRegistry) // Hack, look for a private registry, and decorate the error with the lack of // credentials.  This is heuristic, and really probably could be done better // by talking to the registry API directly from the kubelet here. if explicitRegistry { return fmt.Errorf("image pull failed for %s, this may be because there are no credentials on this request.  details: (%v)", image, err) } return err </ins> } func (p throttledDockerPuller) Pull(image string) error {

I launched a pod forgetting that I had not uploaded my into the node's folder. Of course the image pull failed but the error message was not helpful: e.g. It should finish off the message with "because credentials for are not available to Kubernetes. This may be because blah"


Question: package group import ( <del> "k8s.io/kubernetes/pkg/apis/extensions" </del> <ins> "strings" </ins> "testing" <ins> "k8s.io/kubernetes/pkg/apis/extensions" </ins> ) func TestMustRunAsOptions(t *testing.T) {

Is this a BUG REPORT or FEATURE REQUEST?: This is request for improvements. What happened: When PodSecurityPolicy provider validates or fields of a pod, it shows an error like "999 is not an allowed group" if the group doesn't belong to the range(s). What you expected to happen: It should include to the error message also the allowed values -- the ranges against what we were validating this group. This will simplify user experience and debugging. How to reproduce it (as minimally and precisely as possible): run kubernetes cluster with PSP support create a PSP that allows fsGroup/SupplementalGroups in some ranges grant to a user access to this PSP try to create a pod that has fsGroup/SupplementalGroups outside of the allowed ranges Anything else we need to know?: In order to improve the current behavior, we need to modify function here: Reported-by: /kind help
This looks like a good first issue! My suggestion would be something like . I would also suggest to merge overlaying ranges in the log message, like as , to keep the log as concise as possible. I will look into it.
Maybe something like ? Thank you!
We can use the same approach that strategy uses:
Sure, that is probably the most consistent solution.
/remove-help


Question: cliflag "k8s.io/component-base/cli/flag" "k8s.io/component-base/cli/globalflag" "k8s.io/component-base/configz" <ins> "k8s.io/component-base/metrics/features" </ins> controllersmetrics "k8s.io/component-base/metrics/prometheus/controllers" <ins> "k8s.io/component-base/metrics/prometheus/slis" </ins> "k8s.io/component-base/term" "k8s.io/component-base/version" "k8s.io/component-base/version/verflag"

As per, , wire up metrics/slis for ccm
/assign /sig instrumentation /sig cloud-provider
GitHub didn't allow me to assign the following users: Richabanker. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig instrumentation /sig cloud-provider Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
/remove-triage accepted
/triage accepted


Question: return path, nil }, <del> expectedOutput: false, expectError:    true, </del> <ins> expectedOutput: isRoot,  // ExistsPath success when running as root expectError:    !isRoot, // ExistsPath must fail when running as not-root </ins> }, { name: "relative symlink to existing file",

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT What happened: TestNsenterExistsFile test is failing during make test What you expected to happen: The test should have passed. How to reproduce it (as minimally and precisely as possible): git clone run make test Anything else we need to know?: Fail logs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Others:
/sig storage
If you run the test with root user, It will fail. I have make a fix:
thanks for the patch .. its working.


Question: } } } <ins> func Test_syncService(t *testing.T) { testCases := []struct { oldVirtualServer *utilipvs.VirtualServer svcName          string newVirtualServer *utilipvs.VirtualServer bindAddr         bool }{ { // case 0, old virtual server is same as new virtual server oldVirtualServer: &utilipvs.VirtualServer{ Address:   net.ParseIP("1.2.3.4"), Protocol:  string(api.ProtocolTCP), Port:      80, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, svcName: "foo", newVirtualServer: &utilipvs.VirtualServer{ Address:   net.ParseIP("1.2.3.4"), Protocol:  string(api.ProtocolTCP), Port:      80, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, bindAddr: false, }, { // case 1, old virtual server is different from new virtual server oldVirtualServer: &utilipvs.VirtualServer{ Address:   net.ParseIP("1.2.3.4"), Protocol:  string(api.ProtocolTCP), Port:      8080, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, svcName: "bar", newVirtualServer: &utilipvs.VirtualServer{ Address:   net.ParseIP("1.2.3.4"), Protocol:  string(api.ProtocolTCP), Port:      8080, Scheduler: "rr", Flags:     utilipvs.FlagPersistent, }, bindAddr: false, }, { // case 2, old virtual server is different from new virtual server oldVirtualServer: &utilipvs.VirtualServer{ Address:   net.ParseIP("1.2.3.4"), Protocol:  string(api.ProtocolTCP), Port:      8080, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, svcName: "bar", newVirtualServer: &utilipvs.VirtualServer{ Address:   net.ParseIP("1.2.3.4"), Protocol:  string(api.ProtocolTCP), Port:      8080, Scheduler: "wlc", Flags:     utilipvs.FlagHashed, }, bindAddr: false, }, { // case 3, old virtual server is nil, and create new virtual server oldVirtualServer: nil, svcName:          "baz", newVirtualServer: &utilipvs.VirtualServer{ Address:   net.ParseIP("1.2.3.4"), Protocol:  string(api.ProtocolUDP), Port:      53, Scheduler: "rr", Flags:     utilipvs.FlagHashed, }, bindAddr: true, }, } for i := range testCases { ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) proxier := NewFakeProxier(ipt, ipvs, ipset, nil) if testCases[i].oldVirtualServer != nil { if err := proxier.ipvs.AddVirtualServer(testCases[i].oldVirtualServer); err != nil { t.Errorf("Case [%d], unexpected add IPVS virtual server error: %v", i, err) } } if err := proxier.syncService(testCases[i].svcName, testCases[i].newVirtualServer, testCases[i].bindAddr); err != nil { t.Errorf("Case [%d], unexpected sync IPVS virutal server error: %v", i, err) } // check list, err := proxier.ipvs.GetVirtualServers() if err != nil { t.Errorf("Case [%d], unexpected list IPVS virtual server error: %v", i, err) } if len(list) != 1 { t.Errorf("Case [%d], expect %d virtual servers, got %d", i, 1, len(list)) continue } if !list[0].Equal(testCases[i].newVirtualServer) { t.Errorf("Case [%d], unexpected mismatch, expect: %#v, got: %#v", i, testCases[i].newVirtualServer, list[0]) } } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: IPVS proxier update service too slow, it waits until a sync period. What you expected to happen: Any k8s service update should trigger IPVS virtual service update. How to reproduce it (as minimally and precisely as possible): a k8s service its session affinity type see if ipvs virtual service has changed - NO! Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipvs


Question: shouldFail      bool watchTimeout    time.Duration }{ <ins> // PV agnostic path positive test cases </ins> { <del> testName:        "normal, json file exists", </del> <ins> testName:        "success, json file exists", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        `{"driverName": "csi", "volumeHandle":"project/zone/test-vol1"}`, <del> createPV:        false, </del> stageUnstageSet: true, }, { <del> testName:        "normal, json file doesn't exist -> use PV", </del> <ins> testName:        "stage_unstage not set, PV agnostic path, unmount device is skipped", deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", jsonFile:        `{"driverName":"test-driver","volumeHandle":"test-vol1"}`, stageUnstageSet: false, }, // PV agnostic path negative test cases { testName:        "fail: missing json, fail to retrieve driver and volumeID from globalpath", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        "", <del> createPV:        true, </del> stageUnstageSet: true, <ins> shouldFail:      true, </ins> }, { <del> testName:        "invalid json ->  use PV", </del> <ins> testName:        "fail: invalid json, fail to retrieve driver and volumeID from globalpath", </ins> volID:           "project/zone/test-vol1", <del> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", </del> <ins> deviceMountPath: "plugins/csi/" + generateSha("project/zone/test-vol1") + "/globalmount", </ins> jsonFile:        `{"driverName"}}`, <del> createPV:        true, </del> stageUnstageSet: true, <ins> shouldFail:      true, </ins> }, <ins> // Old style PV based path positive test cases </ins> { <del> testName:        "no json, no PV.volID", volID:           "", </del> <ins> testName:        "success: json file doesn't exist, old style pv based global path -> use PV", volID:           "project/zone/test-vol1", </ins> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", <ins> stageUnstageSet: true, </ins> createPV:        true, <del> shouldFail:      true, </del> }, { <del> testName:        "no json, no PV", </del> <ins> testName:        "success: invalid json file, old style pv based global path -> use PV", </ins> volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", <del> jsonFile:        "", createPV:        false, </del> <ins> jsonFile:        `{"driverName"}}`, </ins> stageUnstageSet: true, <del> shouldFail:      true, </del> <ins> createPV:        true, </ins> }, { <del> testName:        "stage_unstage not set no vars should not fail", </del> <ins> testName:        "stage_unstage not set, PV based path, unmount device is skipped", </ins> deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        `{"driverName":"test-driver","volumeHandle":"test-vol1"}`, stageUnstageSet: false, }, <ins> // Old style PV based path negative test cases { testName:        "fail: json file doesn't exist, old style pv based device path, missing PV", volID:           "project/zone/test-vol1", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", stageUnstageSet: true, shouldFail:      true, }, { testName:        "fail: no json, no PV.volID, old style pv based global path", volID:           "", deviceMountPath: "plugins/csi/pv/test-pv-name/globalmount", jsonFile:        "", shouldFail:      true, }, </ins> } for _, tc := range testCases {

When two CSI PVs point to the same volume in the storage backend, kubelet can't start pods that use these two PVs on the same node. Both PVs have the same VolumeHandle and kubelet is smart enough to recognize that they're the same volume (because they have the same unique volume ID). It calls only once, however, the staging path contains name of one of the PVs, say PV1: One of the CSI volume calls then fails, because it uses PV2 to compute the staging path and that path does not exist. Both pods should just start. Using CSI HostPath volume, PV1 (dynamically provisioned for simplicity): PV2 (basically edited PV1): PVC1, PVC2, Pod1 and Pod2 are trivial, nothing special there. <detailsToday's master, </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage /priority backlog
I suggest that if two volumes have the same unique volume ID (computed from CSI ), they must have also the same global mount path. Kubelet already has code to call NodeStage once and bind-mount it to different pods, regardless how many different PVs they use. These are several issues changing the global mount path: uses PV parsed from the global path to get the PV and to get enough info to call NodeUnstage: However, this parsing is used only when the volume plugin is not able to find/parse its json it has saved during MountDevice: Can we safely assume that the json is always there? the global path in a running cluster is tricky. We can require nodes are drained when kubelet with new global mount path is installed. This way, no CSI volume is mounted to a global path when the new kubelet starts. Minor cluster upgrades (1.23 -1.24) are , also documented in . Or we could fix kubelet to use the old global path when it already exists. This looks fragile to me.

staging dir needs to be unique enough not to conflict with other CSI volumes. It should be based on the driver name + volume handle, but that could be quite long. Can we compute a hash and hope it won't conflict?
I think in-tree driver can handle this case correctly because global mountpath is using unique volume id instead of pv name. With csi-migration, does it break backward compatibility?
We require users to drain nodes before enabling migration or when updating to a Kubernetes version that have migration enabled by default. All volumes should be therefore unmounted.
probably yes this breaks backward compatibility, but since only migration for PD is being done maybe it's more rare? We saw a customer case for this situation with NFS shares using the csi filestore driver. For PD I think it would only come up when trying to ROX a PD volume across namespaces, I'm not sure if that's common.
Capturing some thoughts here in line to understand your we use a staging path like , the kubelet can locate the both the driver name and the volumeId, needed during UnmountDevice call. e.g for pd csi driver which is the sha256{projects/mattcary-saikatroyc-test/zones/us-central1-c/disks/pvc--f97d-43cd-a5f1-}. This means the volume Id needs to persisted (we already do that in today ), so that the volume Id can be retrieved for UnmountDevice call. we decide to make the change at a minor boundary, then a node drain mandate would unstage any volumes before the kubelet is upgraded. this ensures any staged path for csi volume is using the new format 2 If we do not want to enforce a node drain, kubelet can first lookup json data (the reasoning is if this file does not exist, we will not be able to retrieve the volume handle from the new handle anyway), else fallback to try the old format of staging path - . TBD: Need to check mount reference logic. sha(volumeId) is expected to be unique in the cluster, if I understand. (based on the assumption any storage resource provisioned by a given driver is expected to be uniquely identifiable)
/assign
yes, your summary is correct. We persist a json, please make sure it contains anything that NodeUnstage may need.

It seems to me that people prefer option 2.2. We already do to uniquely identify VolumeAttachment. So far nobody complained about any conflict.


Question: } for _, protocol := range protocols { <del> lbRuleName := az.getLoadBalancerRuleName(service, protocol, port.Port, subnet(service)) </del> <ins> lbRuleName := az.getLoadBalancerRuleName(service, protocol, port.Port) </ins> klog.V(2).Infof("reconcileLoadBalancerRule lb name (%s) rule name (%s)", lbName, lbRuleName) transportProto, _, probeProto, err := getProtocolsFromKubernetesProtocol(protocol)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When the specified internal LB uses a subnet with longer name, the LB configuration will fail with following error: What you expected to happen: When constructing the frontend IP configuration name and the load balancing rules name, the subnet name should be truncated if too long. How to reproduce it (as minimally and precisely as possible): Use a subnet that has long name (70 charactors) and create ILB service. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: azure OS (e.g: ): Ubuntu 16.04 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure /assign


Question: if !shouldHold && !affinityHolds { return true, nil } <del> if shouldHold { if !transitionState && !affinityHolds { return true, fmt.Errorf("Affinity should hold but didn't.") } if trackerFulfilled && affinityHolds { return true, nil } </del> <ins> if shouldHold && trackerFulfilled && affinityHolds { return true, nil </ins> } return false, nil }); pollErr != nil {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce-cos-master-slow Which test(s) are failing: [sig-network] Services should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] Since when has it been failing: This test started to be flaky on 11/18 Testgrid link: https://k8s- Reason for failure: Looks to be a pretty straightfowards test not passing: If we look at triage, this test has always been flaky, but it got significantly more flaky (2X failures) on 10/27: Anything else we need to know: This job is non-blocking for 1.13, except if this represents an actual bug in Affinity the 1.13 team needs to know ASAP. This failure may or may not be related to . It looks like this test was disabled for large clusters, but maybe the problem wasn't the cluster size. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /sig network /priority important-soon attn:
FWIW this seems to be a long standing issue, atleast as far bck as 1.12 timeframe
Sure. But the failures have become 3X as frequent, so I still want a sign-off by sig-network.
cc
Ack. Will work on this.
we plan to lift code freeze for 1.13 tomorrow,11/28 5pm PST. If there is a new regression in 1.13 thats increased the flakes and needs addressing we need to get the fix in asap. Can you please let us know if this is a blocker for 1.13 by tomorrow, 11/28, morning 10am PST ?
Great observation. I checked on the commits around that date but wasn't able to find any change that may cause regression. Hasn't been able to reproduce the same failure locally as well.
I just noticed that the same test is flaking a lot more on 1.12 branches as well https://k8s-
Good catch on that, so the recent increase of flakiness likely is not caused by 1.13 changes so we shouldn't block release on this either. cc
But we will be looking deeper into the root cause.
Several observations: It only happens when is false.

When it's true, the test does more stuff before actually check service session affinity. It doesn't happen when executing in pod, probably due to the same reason above, the test start test pod before actually testing service. I wrote a script which does the same thing in my cluster of version 1.11.3-gke.18 in us-central1-a. cannot reproduce it. The hypothesis is that the service session affinity spec takes some time to reconcile after the service is created. It could happen either on the node or on the GCLB. Since I haven't been able to reproduce it's hard to tell. But it's more likely on GCLB side, even it could be regional issue. Looks like a sleep of seconds will deflake it. but it's just hiding issues. I would rather open another issue to track the real issue.
The most recent failure is here: Basically what happened is GCLB doesn't honor session affinity and still sends the request to different VMs. The log shows that 3 requests are routed to 3 different VMs. Dec 19 07:40:58.094: INFO: [service-fsn65 service-bf8nc] Dec 19 07:40:58.094: INFO: Affinity should hold but didn't. Dec 19 07:40:58.196: INFO: [pod,node] pairs: [{Pod:service-bf8nc Node:bootstrap-e2e-minion-group-f3nb} {Pod:service-fsn65 Node:bootstrap-e2e-minion-group-9sp7} {Pod:service-rnrcw Node:bootstrap-e2e-minion-group-m5rj}]; err: <nil
More recent failures: Logs:
Hello, could this issue be prioritized to help with stabilizing the release blocking upgrade tests? They're currently flaking ~23% of the time in () ~30% of the time in ()
/assign can you take a look
GitHub didn't allow me to assign the following users: take, a, look, can, you. Note that only and repo collaborators can be assigned and that issues/PRs can only have 10 assignees at the same time.

For more information please see <detailsIn response to : /assign can you take a look Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am preparing a PR, basically allow Affinity some time to be configured, instead of failing out instantly.
Thanks! Will keep an eye out for it.
The test is no longer flaky: https://k8s- Thanks everyone!
Thanks! Will keep an eye on it for a few days, on the following dashboards:
It's stabler now, but still flaking once/twice a day, e.g. in . - Is this the same issue as you looked at? Is there more to be done here? /reopen
Reopened this issue. <detailsIn response to : It's stabler now, but still flaking once/twice a day, e.g. in . Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
we should probably cherrypick into 1.13 in case of deflaking upgrade/downgrade job, because those jobs would run 1.13 tests against 1.14 (upgraded) cluster or 1.13 (downgraded) cluster. FYI we've hit similar situation for DNS test (ref ).
Good point. Cherrypick PR prepared.
14: <img width="1110" alt="Screen Shot 2019-03-16 at 12 11 07 PM" src="https://user-"13: <img width="1091" alt="Screen Shot 2019-03-16 at 12 11 22 PM" src="https://user-"Do we want to cherry-pick this back further?
Thanks for the nice graph. Look like the fix works. I'll create cherrypick PRs for 1.11 and 1.12 right away.
AFAICT this has been significantly improved: ! Closing this and thanks to /close
Closing this issue. <detailsIn response to : AFAICT this has been significantly improved: ! /close Instructions for interacting with me using PR comments are available .


Question: podName := "pod-dualstack-host-ips" <del> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   podName, Labels: map[string]string{"test": "dualstack-host-ips"}, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "dualstack-host-ips", Image: imageutils.GetE2EImage(imageutils.Agnhost), }, }, HostNetwork: true, }, } </del> <ins> pod := genPodHostIPs(podName+string(uuid.NewUUID()), true) </ins> ginkgo.By("submitting the pod to kubernetes") podClient := e2epod.NewPodClient(f)

4 other Test Failures in ci-crio-cgroupv1-node-e2e-features. Test has failed 61 times (no passing result found). Earliest failure found at commit/ (Thu Oct 19 21:41 PDT). /cc { failed [FAILED] Error creating Pod: pods "pod-dualstack-host-ips9ddb9810-134c-4d35-a0c6-" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true) In [It] at: @ 10/23/23 01:38:06.024 } No response_ /sig network node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
is also failing on these tests
the test has a bug, it checks the contrary
, let's start over, 1 commit for the e2e test, other for the promotion, and let's validate everything is alright before merging


Question: return suggestedDelay, utilerrors.NewAggregate(errs) } <del> func toFlowSchemaApplyConfiguration(fsUpdate fsStatusUpdate) *flowcontrolapplyconfiguration.FlowSchemaApplyConfiguration { condition := flowcontrolapplyconfiguration.FlowSchemaCondition(). WithType(fsUpdate.condition.Type). WithStatus(fsUpdate.condition.Status). WithReason(fsUpdate.condition.Reason). WithLastTransitionTime(fsUpdate.condition.LastTransitionTime). WithMessage(fsUpdate.condition.Message) return flowcontrolapplyconfiguration.FlowSchema(fsUpdate.flowSchema.Name). WithStatus(flowcontrolapplyconfiguration.FlowSchemaStatus(). WithConditions(condition), ) </del> <ins> // makeFlowSchemaConditionPatch takes in a condition and returns the patch status as a json. func makeFlowSchemaConditionPatch(condition flowcontrol.FlowSchemaCondition) ([]byte, error) { o := struct { Status flowcontrol.FlowSchemaStatus `json:"status"` }{ Status: flowcontrol.FlowSchemaStatus{ Conditions: []flowcontrol.FlowSchemaCondition{ condition, }, }, } return json.Marshal(o) </ins> } // shouldDelayUpdate checks to see if a flowschema has been updated too often and returns true if a delay is needed.

A lot of different routines failing just when setting up an apiserver. Example: <details</detailsNo errors The test still passes, but those errors seem very worrisome <detailsmaster </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /sig testing cc
yeah, the framework has some long standing tech debt /cc
all the panics I see are:
looks like again, due to
Looks like some integration tests must not be enabling SSA.
yeah, a lot don't, because it requires openapi, which also isn't universally enabled in integration. it is really expensive at API server startup, and when we tried enabling it universally in integration it blew up runtimes and integration timed out super frequently
that's what is still open for
/triage accepted


Question: serverVersion = flag.Bool("server_version", false, "Print the server's version number.") preventSkew   = flag.Bool("expect_version_match", false, "Fail if server's version doesn't match own version.") httpServer    = flag.String("h", "", "The host to connect to.") <del> config        = flag.String("c", "", "Path to the config file.") </del> <ins> config        = flag.String("c", "", "Path or URL to the config file.") </ins> selector      = flag.String("l", "", "Selector (label query) to use for listing") updatePeriod  = flag.Duration("u", 60*time.Second, "Update interval period") portSpec      = flag.String("p", "", "The port spec, comma-separated list of <external>:<internal>,...")

The kubecfg -c option is used to pass a file pointer to a representation of a model object on disk (pod, replicationController, service) for use by the operation. It would be nice if I could do the following: Where kubecfg sees the http:// and does a network operation to fetch resource for me. This would let users who want to build tutorials on how to use kubernetes to avoid steps to manually download the files locally via curl and unzip, etc. and would also let scripters that use the kubecfg client to avoid needing write access on the terminal where they run kubecfg to perform basic actions.


Question: } } } <ins> if isDeletion { // cachedService is not reliable here as // deleting cache is the last step of federation service deletion _, err := fedClient.Core().Services(cachedService.lastState.Namespace).Get(cachedService.lastState.Name) // rebuild service if federation service still exists if err == nil || !errors.IsNotFound(err) { return sc.ensureClusterService(cachedService, clusterName, cachedService.appliedState, clusterCache.clientset) } } </ins> return nil }

As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.


Question: // IsMountPoint determines whether a path is a mountpoint by calling findmnt // in the host's root mount namespace. <del> func (*NsenterMounter) IsMountPoint(file string) (bool, error) { </del> <ins> func (n *NsenterMounter) IsMountPoint(file string) (bool, error) { </ins> file, err := filepath.Abs(file) if err != nil { return false, err } <del> args := []string{"--mount=/rootfs/proc/1/ns/mnt", "/usr/bin/findmnt", "-o", "target", "--noheadings", "--target", file} </del> <ins> args := []string{"--mount=/rootfs/proc/1/ns/mnt", "--", n.absHostPath("findmnt"), "-o", "target", "--noheadings", "--target", file} </ins> glog.V(5).Infof("findmnt command: %v %v", nsenterPath, args) exec := exec.New()

What happened: Windows documentation claims: V1.VolumeMount.mountPropagation - only MountPropagationHostToContainer is available. However, the internals suggest otherwise; is always returned on Windows: Either the docs should be corrected, or the Kubelet should be updated, or we should have a comment on why this does in fact enable MountPropagationHostToContainer on Windows. /cc /sig windows
/cc
/cc
I think the doc is not accurate. The doc claims the same thing. For MountPropagationHostToContainer option, we can test it whether it is working or not.
i created two PRs to address the documentation gap. we will also test this and get back to this thread.
For 1.15 we may need to update Kubelet code to allow HostToContainer.


Question: JSONPatchMaxCopyBytes:       10 * 1024 * 1024, MaxRequestBodyBytes:         -10 * 1024 * 1024, }, <del> expectErr: "--max-resource-write-bytes can not be negative value", </del> <ins> expectErr: "ServerRunOptions.MaxRequestBodyBytes can not be negative value", </ins> }, { name: "Test when LivezGracePeriod is negative value",

kube-apiserver has no --max-resource-write-bytes & --json-patch-max-copy-bytes option. Correct the error message to not use the "--max-resource-write-bytes" & "--json-patch-max-copy-bytes" string. Check the source code: No response v1.19 ~ v1.23 eg: </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery /area apiserver /kind cleanup /assign FYI:
/cc /triage accepted


Question: } func (plugin *cniNetworkPlugin) TearDownPod(namespace string, name string, id kubecontainer.ContainerID) error { <ins> if err := plugin.checkInitialized(); err != nil { return err } </ins> netnsPath, err := plugin.host.GetRuntime().GetNetNS(id) if err != nil { return fmt.Errorf("CNI failed to retrieve network namespace path: %v", err) } <del> return plugin.defaultNetwork.deleteFromNetwork(name, namespace, id, netnsPath) </del> <ins> return plugin.getDefaultNetwork().deleteFromNetwork(name, namespace, id, netnsPath) </ins> } // TODO: Use the addToNetwork function to obtain the IP of the Pod. That will assume idempotent ADD call to the plugin.

Host network pods should be able to be created before the container network exists. Pod syncs for pods on the container network should fail until the container network is configured. cc
ref:
I think this is a thing we should make working before v1.4, so adding the milestone. Complain if you disagree. Also, does volunteer to make a PR? :)
Going to make a first pass at this in
Great!
/cc


Question: recorder      events.EventRecorder nodeRef       *v1.ObjectReference <del> lastUpdated atomic.Value lastQueued  atomic.Value </del> <ins> lastUpdated         atomic.Value oldestPendingQueued atomic.Value </ins> } // NewProxierHealthServer returns a proxier health http server.

We upgraded iptables on one of our hosts, and kube-proxy began logging errors about being unable to acquire the . However, we didn't notice this, because kube-proxy's health check () was still passing (and that's generally what we look at as our sign that something is amiss.) As a result, kube-proxy ran for days without being able to update iptables rules, eventually causing an outage for one of our services. We expected that kube-proxy's health check would fail if it was not able to interface with iptables. Our system was running Debian stretch, running kube-proxy in iptables mode with systemd, with iptables version 1.6.0. Given that starting configuration, the repro case is: iptables to 1.6.2. that kube-proxy begins emitting logs saying that it cannot update iptables because another process has the . that kube-proxy's health check endpoint () does not return an error. Example health check output: <details</details<details</details<detailsNone </details<details</details<detailsThe hard way </details<detailsDocker version 19.03.8, build </details<detailsCalico v3.18.4 </details
/sig network
you upgraded iptables just one node ? what is the version of iptables on other nodes ?
does it means that is not able to never get the lock or is just that sometimes is able to get it?
Yeah we upgraded on just one node at first. Like my original bug report says, we started at 1.6.0, so all the other nodes had 1.6.0. it never gets the lock. You have to restart kube-proxy so it can go through the iptables version detection routine again and change its behavior to the 1.6.2 logic branch. That's fine, it's whatever, but the health check doesn't fail even though kube-proxy can't do its job, which is a bug in my opinion. Also yes, we checked to see if some other process had the lock. It didn't. It's something to do with changes in how iptables actually works after the upgrade to 1.6.2, specifically. I believe this launchpad issue is relevant:
the logic of the healthcheck should fail if is not able to process the updates, right?

I this to the description as well, but here is an example response during this failure mode: And here are what the logs look like: Running gives no output.
This is interesting the and values are the same. This is the healthz logic, it will be unhealthy if the difference between both is greater than the health timeout. is it possible to share the entire kube-proxy log ? I want to discard something
to save me some time, could you please tell me what sort of messages you are looking for? I would rather go search splunk for them than to set up a repro case again.
just the first 1000 lines since kube-proxy starts should be enough
The JSON is kind of meaningless these days; if the proxier thinks it's healthy, it will just write out the current time as both and . It's only when it thinks its unhealthy that it writes out the real time. See I'm not sure how this could happen...
I'm afraid I'm not at liberty to give you 1000 lines of logs without some more targeted context. Company policy and whatnot. What I will tell you is I put the proxy up to and reproduced the issue and pulled those logs, and if you tell me the kind of messages you are looking for, I can pull the (redacted) context around them. What I can give you now, even without being directed, is the more verbose version of what I posted before, i.e., what happens around the time that the error gets logged: <detailsFollowed by the whole list of iptables rules. Then </detailsI can also give you the startup arguments: <details</detailsAnd finally, I can give you these lines from the iptables code, the health check getting registered, and a couple other startup items: <details</detailsAny more than this, though, and I'll have to ask you to tell me something to grep for.
First thing to check is that the healthcheck actually works EVER. Given the same timestamp (we should fix that?) it thinks it is healthy.
/assign
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
A situation: when there is a service or endpoint is constantly updated (which will update the ): so there is a high probability that the difference between the and the is less than the , so this will be considered healthy If you reduce ( iptables-sync-period * 2, the defaults value of flag is 30s), you may see response code is 503, but in fact, it is not easy to estimate how much it should be reduced. so As for case 3, I think there are something that can be optimized.
Oh! That's wrong. We don't care about the time of the most-recently queued update, we care about the time of the oldest queued update that was after the last . So rather than having and updating it every time gets called, we should have a variable something like , and can set it to 0, and should only set it to if it was previously 0. In fact, we don't even need . Kube-proxy would be healthy if or .
Thanks I see, I'd fix it, can I take this one?
sure /assign


Question: func RestartKubeProxy(host string) error { // TODO: Make it work for all providers. if !ProviderIs("gce", "gke", "aws") { <del> return fmt.Errorf("unsupported provider: %s", TestContext.Provider) </del> <ins> return fmt.Errorf("unsupported provider for RestartKubeProxy: %s", TestContext.Provider) </ins> } // kubelet will restart the kube-proxy since it's running in a static pod Logf("Killing kube-proxy on node %v", host)

is failing with "framework.RestartControllerManager() = unsupported provider: gke; want nil" /kind failing-test
/sig network
Introduced in , should be fixed by