Question: firstManifestURL := ServeCachedManifestFile(testPodSpecFile) secondManifestURL := ServeCachedManifestFile(testManifestFile) <del> apiServerURL, configFilePath := startComponents(firstManifestURL, secondManifestURL, apiVersion) </del> <ins> apiServerURL, _ := startComponents(firstManifestURL, secondManifestURL, apiVersion) </ins> // Ok. we're good to go. glog.Infof("API Server started on %s", apiServerURL)

Probably not the same as ? To for triage
Shouldn't this fixed by ?
this has nothing to do with . kubelet is trying to create the mirror pod over and over again because it still hasn't observed the mirror pod from watch. It is probably the same as # .
I thought they might be different because this one has a bunch of "failed creating ... already exists" messages in it, but doesn't.
It is essentially the same problem -- the watch was delayed so kubelet kept seeing the old snapshot of the pods where the mirror didn't exist. That's why it's flaky and only happens on shippable (which I suspect has poorer performance) :( This becomes worse every time a new test is . I think we need to limit the number of concurrent tests in
limits the number of concurrent tests on travis/shippable to 4. Let's see if this helps reduce the timeout flakes.
Another example:
Here is yet another example: Retitled the issue since limiting the concurrent tests doesn't work. It could be that the server is heavily loaded and the test timed out, or there was a delay/drop in kubelet's pod watch stream.
Now example:
If, for whatever reason, a watch update for pod creation is dropped, kubelet would need to wait until re-listing to notice that. This would certainly cause the test to time out. would it be useful to shorten the re-listing interval for tests like this?
I am not quite sure how this works--let's chat about it tomorrow?
From , we can see that kubelet was lightly loaded as the update channel was mostly empty while waiting. This corroborates the theory that watch might be at fault.
cc: quinton-hoole
The WIP reflector benchmark () may shed some light on this issue once it's integrated with the continuous e2e performance suite. I will keep this issue open for now to gather more data (since this is not reproducible locally at all), and to see if there are other potential causes.
The PR above disables the static pod test temporarily so that we can track down the root cause without spamming people with the failures. Note that other tests in the same file () may still trigger the flaky failure, though less frequently.
Can this be closed yet?
No, integration test is still flaky. See for example.

Many tests in rely on checking the pod status with a preset timeout. I disabled one test which failed most often, but other tests still fail from time to time.
The issue has been inactive for months, and there have not been any new reports. Closing.


Question: OUT_DIR?=_output HYPERKUBE_BIN?=$(shell pwd)/../../../$(OUT_DIR)/dockerized/bin/linux/$(ARCH)/hyperkube <del> BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.10.1 </del> <ins> BASEIMAGE=k8s.gcr.io/debian-hyperkube-base-$(ARCH):0.10.2 </ins> TEMP_DIR:=$(shell mktemp -d -t hyperkubeXXXXXX) all: build

What happened: Kube-proxy ipvs mode CI failed http://k8s- But actually all test are passed in my local-up environment with the newest code. According to ci log, those pr might be related: seems like the most probable, Anyway, looking for help.
Friendly ping , Please take a look, looking forward for your advise.
thanks for the report! /sig network /sig gcp /kind failing-test
/area ipvs
Update: Seems like staging- image do have some problem, command will return err. Not likely to be ipset issue.
Suggestions?
The version is exactly the same in both versions of the image (v10 and v10.1). Where did you find that error message?
List of package differences (given by ) between v10 and v10.1:
Thanks, the error was found in ipvs ci environment kube-proxy log. And I also get this error in my local environment with
Maybe we got some kind of incompatible issue with ipset specific version.
It seems like the removal of is what's causing this failure, which appears to be due to depending on instead of . I'm not sure what caused that to switch.
ok, I think I have an idea of what happened here: was originally built against . At some point was built against what was then , i.e. pulling in the packages from , which included (and ). Later (probably due to security issues), we rebuilt , which by this point was now based on . In this release, they apparently switched to installing by default, which did not include a dependency on . We apparently did not rebuild , though I'm not sure why not. As a result, it was still based on the older debian release. By the time we built , had been released, and so was similarly based on and was thus missing . tl;dr: need to add a dependency on in apparently, and should build/push a (or maybe ). should probably be more careful with these "simple rebases", since they can unexpectedly change what packages are installed. Some automation to do would help here. (cc do you think this is a kubernetes 1.12-blocking issue?
/assign
Thanks a lot. Though I'm not really understand, but glad we found the root cause.

And I think this gonna block 1.12 release, ipvs feature already GA in 1.11, we need make sure this feature can work normally.
for thoughts.
Do you think this is a kubernetes 1.12-blocking issue?
If this causing core IPVS functionality to be broken, which it looks like it is, then I think this should block the 1.12 release Sent from my iPhone
/milestone v1.12 Will treat as release blocking now until we hear otherwise from SIG-Network and SIG-GCP
fix PR for the s is ; we'll need one more PR after that's merged to update the Makefiles to use the new images.
First PR has merged.
Thanks. Working on promoting those images to , though that probably won't happen till Tuesday PT.
... or maybe sooner than that, because is an overachiever.
Second PR:
You're the one up working on promoting images at 9pm :-) I'm just trying to set my dev environment up again and saw this going on.
I can't keep up with the 3 of you!!! - -- An old man
The CI working well again hours ago, thanks all of you, really appreciate it. : )
Awesome. We might want to get better monitoring on this job if IPVS is GA, since it took several weeks from the original PR merged before this issue was opened.
Anyone willing to monitor can PR email alerts per config here: Please consider doing so! It's ~hard~ impossible to track all of testgrid, we really need SIGs to step up on this!
Definitely make sense, I will raise a pr to do this work.
Thank you! Lion-Wei <:


Question: var socket utilproxy.Closeable switch lp.Protocol { case "tcp": <del> listener, err := net.Listen("tcp", net.JoinHostPort(lp.IP, strconv.Itoa(lp.Port))) </del> <ins> network := "tcp4" if isIPv6 { network = "tcp6" } listener, err := net.Listen(network, net.JoinHostPort(lp.IP, strconv.Itoa(lp.Port))) </ins> if err != nil { return nil, err } socket = listener case "udp": <del> addr, err := net.ResolveUDPAddr("udp", net.JoinHostPort(lp.IP, strconv.Itoa(lp.Port))) </del> <ins> network := "udp4" if isIPv6 { network = "udp6" } addr, err := net.ResolveUDPAddr(network, net.JoinHostPort(lp.IP, strconv.Itoa(lp.Port))) </ins> if err != nil { return nil, err } <del> conn, err := net.ListenUDP("udp", addr) </del> <ins> conn, err := net.ListenUDP(network, addr) </ins> if err != nil { return nil, err }

/kind bug What happened: When creating a node port application (in this case the example guestbook) on a host server that is dual stacked and has AAAA and A DNS entries, the node port on the v6 address is open to kube-proxy and hangs. What you expected to happen: The v6 address would refuse the connection and the client would fall back onto the v4 address which would connect correctly. How to reproduce it (as minimally and precisely as possible): kubeadm init with the above, and join in some workers. Then install the guestbook application Obtain the node port with and telnet to one of the servers on that port from a dual stacked client. The client will connect and refuse to respond to 'GET /' telnet directly to the IPv4 address of the server on the port and all works as expected. Anything else we need to know?: kube-proxy is noted as listening on the port. Environment: Kubernetes version (use ): ubuntu kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:53:20Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-07-17T18:43:26Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Brightbox OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS Kernel (e.g. ): Linux srv-jidtj 4.15.0-29-generic -Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm 1.11 Others:
/sig network
Looks like kube-proxy in ipvs mode listens on the nodeport ports to ensure they're not in use by something else. It just binds and listens, it doesn't serve requests.

It binds on both ipv4 and ipv6 but never expects to receive connections because the iptables/ipvs rules intercept them, but on ipv4 clusters there is nothing stopping ipv6 connections coming in. The comment there does admit that only a bind is necessary to do this, but doing a full listen just made e2e tests easier. /cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen This now causes conformance failure on e2e tests due to the promotion of the Nodeport test in
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Still an issue on 1.16 when running in standard IPv4 mode
As I understand this thread, I have the same problem. I thought that it is a node configuration or some kubernetes flag that I didn't set but it seems it ain't.

I ask already on SO: Environment Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:36:53Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-10-02T16:51:36Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: On premise, on virtual machines using hyper-v OS (e.g. from /etc/os-release): NAME="Ubuntu" VERSION="18.04.3 LTS (Bionic Beaver)" Kernel (e.g. uname -a): Linux kubernetes 5.0.0-29-generic # 31~18.04.1-Ubuntu SMP Thu Sep 12 18:29:21 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm version: &version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:34:01Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
/lifecycle frozen
/assign bowei
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Looking through the comments, this looks like a bug that could have a targeted fix. /remove-triage unresolved /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Looking through the comments, this looks like a bug that could have a targeted fix.

/good-first-issue /help-wanted </details
I will attempt to solve this issue.
any success or progress? If you left this issue I would like to give it a go as a first time contributor. Thanks, Yuval
I'm still working on this as my first issue, and planning to produce a solution by 20th.
I wasn't able to tackle the issue and will not continue trying, feel free to take it.
/assign yuvalsteuer
Still, open -- anyone actively working on this? I would like to attempt if not. :) /assign rkc007
OK, again still open, doesn't look like anyone is working on this. I'd like to try it. The fix that occurs to me is simply to only listen to IPv6 addresses if the cluster is IPv6. If the e2e tests fail in this case (from looking at I don't think they will) then maybe they need to be updated. Does that make sense? What I'm not sure about is if we should do nothing with IPv6 ports or bind (to claim the port) but not listen.
/assign michaelbeaumont


Question: <del> check process kube-apiserver with pidfile /var/run/kube-apiserver.pid group kube-apiserver start program = "/etc/init.d/kube-apiserver start" stop program = "/etc/init.d/kube-apiserver stop" if failed host 127.0.0.1 port 8080 protocol http request "/index.html" then restart </del>

cluster/saltbase/salt/monit/kube-apiserver appears not to be referenced by and so can be deleted, I think
/cc
created PR thanks!


Question: package util import ( <ins> "net" </ins> "regexp" )

When spinning up VMs w/ kube, I attempted to use but this fails. ironically, 127.0.0.1 succeeds (internally you can then access a kube service via , for example, but you cannot do this if you try to bind to 0.0.0.0. binding works to 127.0.0.1, but is a horrible idea (since this IP wont be reachable by the outside world), so in that case kube should print a clear message on why this is the case and fail. binding to 0.0.0.0 fails right out of the bat, so in this case, also, kube should fail immediately. To be clear, im suggesting that these should fail at the service submission layer. Possibly a module for validating logic of submitted services/pods , which extends syntactical validation is in order if one doesn't already exist?
Please also see
Since this is very similar to I'm gonna take this.


Question: if err != nil { return ControllerManagerMetrics{}, err } <del> output, err := g.getMetricsFromPod(g.client, metricsProxyPod, metav1.NamespaceSystem, kubeControllerManagerPort) if err != nil { return ControllerManagerMetrics{}, err } </del> return parseControllerManagerMetrics(output) }

! /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig instrumentation /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The flake happened in before.
This is fixed. !
/reopen This is still failing. Sample run:
Reopened this issue. <detailsIn response to : /reopen Sample run: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
The error message in makes me think the apiserver is not available during that test run: . Can someone link to the frequency graphs to see if the problem is widespread?
got it:
I don't think I was right above. I think the "the server is currently unable to handle the request" is an error coming from the proxy. The proxy was around the time this issue was filed (), so it seems likely to be related to it. can you take a look?
This is a known issue, and is working on it . Now that is being blocked by the debugging handler problem, could we merge to mitigate this problem first? cc
I would prefer to reach a decision about enabling the debug handlers, but if that's not possible, then merging PR as mitigation might make sense.
Think I hit this also.
It appears to still be failing based on
Why rebase? Any new test run will be based on merge with current k/k master and thus contain
<img width="1111" alt="Screen Shot 2021-05-28 at 13 04 52" src="https://user-" looks like it has become more stable now.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" "k8s.io/kubernetes/test/e2e/manifest" <ins> e2ereporters "k8s.io/kubernetes/test/e2e/reporters" </ins> testutils "k8s.io/kubernetes/test/utils" utilnet "k8s.io/utils/net"

What would you like to be : I would like a method of seeing more clear data while tests are running, e.g.: How many tests will be run, have been run, have passed, have failed, etc. Name of the tests passed/failed/skipped Why is this needed: While a run is going we only have the output from stdout to investigate. There is a LOT of noise in those logs which is great for debugging after the fact, but difficult to consume most of the time. you can't tell how many tests have been run or get a sense of how much time is left you cant easily get a list of all tests that have failed so far My proposal Adding a new, custom ginkgo reporter that will post the data to a given URL (passed via a flag) if the URL isn't set, the reporter is a noop and doesn't waste time/resources Use cases: A user runs the tests dumps stdout to a file for debugging later, meanwhile they run a local server that will just echo the status updates it gets. At any point they look at the output they can get good data about what would be run, how far along it is, how successful it has been so far. Using the conformance test image and Sonobuoy, the Sonobuoy worker can act as the server, taking those status updates and reporting them to the aggregator. This would allow Sonobuoy to present that information to users on demand or stream it depending on the context. The work of adding this is pretty minimal and is clearly useful. In the case of Sonobuoy; this has been the most requested feature for a long time showing that people in the community really do want to have a better high-level picture of the progress of these tests. I think emitting the data to an endpoint like this is a flexible implementation that can help any consumer of the tests (not just Sonobuoy).
/sig testing /area test /area test-framework
One notification URL should work fine. Can you give me details on the data model for the status data to be published?
That is one, slightly TBD area. The reporter in the test framework has a bunch of data and I'm not sure what we exactly Sonobuoy may want from any given plugin (not just the e2e plugin).

I don't want this implementation to be overly Sonobuoy specific or overly e2e specific. Will update when something more clear is pinned down.
Brought this up in the testing commons meeting and it was agreed that this approach seemed reasonable and that the customReporters seemed like the only way to generate this data. In addition, it was requested that we find a way to also emit this (or some variation) into stdout/logs so that it is easier for a human to navigate those verbose logs.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: INSTANCE_PREFIX="${KUBE_AWS_INSTANCE_PREFIX:-kubernetes}" CLUSTER_ID=${INSTANCE_PREFIX} AWS_SSH_KEY=${AWS_SSH_KEY:-$HOME/.ssh/kube_aws_rsa} <ins> CONFIG_CONTEXT="${KUBE_CONFIG_CONTEXT:-aws_${INSTANCE_PREFIX}}" </ins> IAM_PROFILE_MASTER="kubernetes-master" IAM_PROFILE_NODE="kubernetes-minion"

I've run to create a cluster, which resulted in a cluster and user being created in kube-config. Now I want to run again with different parameters, to create a second cluster in a different AWS account. I would like to use a different name for the cluster and user for this cluster. From what I can see from the , there seems to be no way to completely override the name used. I understand that I can use a separate kube-config file, but I don't want to do that. I want to use the same kube-config file, with multiple clusters and users, all handled by .
Manually overwriting file to set CONTEXT to a different value works to achieve the effect that I want. This value should be allowed to be set externally so that we don't have to resort to manually editing the file for each cluster.
Fixed in


Question: return volumetypes.GeneratedOperations{ OperationFunc:     mountVolumeFunc, EventRecorderFunc: eventRecorderFunc, <del> CompleteFunc:      util.OperationCompleteHook(util.GetFullQualifiedPluginNameForVolume(volumePlugin.GetPluginName(), volumeToMount.VolumeSpec), "volume_mount"), }, nil </del> <ins> CompleteFunc:      util.OperationCompleteHook(util.GetFullQualifiedPluginNameForVolume(volumePluginName, volumeToMount.VolumeSpec), "volume_mount"), } </ins> } func (og *operationGenerator) resizeFileSystem(volumeToMount VolumeToMount, devicePath, deviceMountPath, pluginName string) (simpleErr, detailedErr error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In this test, volume mount failed because the driver-registrar took a minute to come up. However, it seems like volume mount operation is retrying without any sort of backoff. This message appears in 400+ times in less than a minute. What you expected to happen: Operation executor should not be retrying operations so frequently. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
I'll take a look at this. /assign
Problem is that reconciler depends on operation executor to do backoff. In the normal case operation executor returns a method that does the actual work. Any errors by that method result in exponential backoff. But in this case, the error is coming from the code creating the mount method which is not handled by the back off logic, and is called again and again by the reconciler resulting in log and event spam. This didn't use to be a problem because initialization errors were uncommon (plugins were built in), but with CSI the driver may not exist and these initialization errors are expected. I'll look in to cleaning this up.
Is this something that will affect users in 1.13 and needs CPing?
This will result in kubelet log spam and possibly kubernetes event spam. It happens when someone creates a pod referencing a CSI volume before the CSI driver for that volume is installed. Which happens frequently in E2E tests because they install/remove driver with every test. But should be rare in real world usage since most users are likely to install a driver first before using it.
If release team is worried about this, we could create a quick PR to suppress logging/event generation of this these errors. Otherwise we plan to address this more holistically for 1.14 and 1.13.1.


Question: } return TestContext.GCEUpgradeScript } <ins> func waitForSSHTunnels() { Logf("Waiting for SSH tunnels to establish") RunKubectl("run", "ssh-tunnel-test", "--image=gcr.io/google_containers/busybox:1.24", "--restart=Never", "--command", "--", "echo", "Hello") defer RunKubectl("delete", "pod", "ssh-tunnel-test") // allow up to a minute for new ssh tunnels to establish wait.PollImmediate(5*time.Second, time.Minute, func() (bool, error) { _, err := RunKubectl("logs", "ssh-tunnel-test") return err == nil, nil }) } </ins>

Upgrade fails with: Error from: https://k8s- Related failures: https://k8s- https://k8s-
/cc
might require how do the GKE tests interact with the defaults you set up?
Looks like are pretty isolated to upgrade tests.
Also we didn't start seeing this until the ABAC default change merged so I think change might help
it will definitely help with the GCE jobs mentioned in the description of
I haven't looked at the logs, so sorry if this is just noise, but the symptom (couldn't get logs b/c pod doesn't exist) looks very similar to what happened when the kubectl taint e2e tests were running in parallel instead of serial.
Judging by the comment and digging into the history , this used to be wrapped in a retry loop. It was removed in PR
it looks like this is limited to gke jobs in the upgrade tests. since getting pod logs depends on master-node communications, it would be affected by the SSH tunnels issue ()
Not blocking under guidance


Question: run(ctx, filterInitializers(controllerInitializers, leaderMigrator.FilterFunc, leadermigration.ControllerMigrated)) }, OnStoppedLeading: func() { <del> klog.Fatalf("migration leaderelection lost") </del> <ins> klog.ErrorS(nil, "migration leaderelection lost") klog.FlushAndExit(klog.ExitFlushTimeout, 1) </ins> }, }) }

kube-controller-manager yields whole stack trace when it loses leaderelection. kube-controller-manager to only log and to exit (without logging 17K lines of stack trace that is not useful for anyone). kube-controller-manager yields around 17K lines of stack trace (omitted in the above example). No response <detailsv1.20.12 </details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/assign /triage accepted
has Fatalf always triggered a stack dump of all goroutines?
/cc
Seems like does a dump of all goroutines Could we and then maybe? /cc in case you have thoughts about this
If I read the right, it only printed the current goroutine to stderr. Dumping everything seems to have been later. make sense to me.
Dumping of all goroutines was in
I have similar doubts. I want to fix it. /assign
kube-scheduler seems to have a similar issue (but a shorter stack trace)
When replacing (useful because it doesn't do structured logging, not just because of the stack trace problem), remember to use + the new . Updated migration instructions are pending in There's also - reverting the klog change might be a faster way to address this.


Question: gvk := obj.GetObjectKind().GroupVersionKind() t := c.parser.Type(gvk) if t == nil { <del> return nil, fmt.Errorf("no corresponding type for %v", gvk) </del> <ins> return nil, newNoCorrespondingTypeError(gvk) </ins> } return t.FromUnstructured(u) }

Found by failing TestApplyConvertsManagedFieldsVersion test in () It seems that conversion depends on schema info that is collected from served API versions only. However, managedFields stores data in etcd from REST versions served in previous releases which may no longer be served. That means that we have to be able to deal with managedFields persisted via API calls to versions that are no longer served, but which we still have and understand conversions for. /cc /sig api-machinery /wg apply /priority important-soon
This may be fixed in a more recent version of structured-merge-diff. I'll look into it
/assign
At one point we were using the types registered to the scheme to generate the openAPI definitions used by apply, but we switched to using the installed endpoints instead, because it was less invasive (We needed to expose which types are convertible to which other types, and also a way to skip generating openAPI for internal types). We might want to revisit the first approach, given this use case.


Question: // AttachDisk attaches a vhd to vm. The vhd must exist, can be identified by diskName, diskURI. // return (lun, error) func (c *controllerCommon) AttachDisk(isManagedDisk bool, diskName, diskURI string, nodeName types.NodeName, cachingMode compute.CachingTypes) (int32, error) { <ins> if isManagedDisk { diskName := path.Base(diskURI) resourceGroup, err := getResourceGroupFromDiskURI(diskURI) if err != nil { return -1, err } ctx, cancel := getContextWithCancel() defer cancel() disk, err := c.cloud.DisksClient.Get(ctx, resourceGroup, diskName) if err != nil { return -1, err } if disk.ManagedBy != nil { attachErr := fmt.Sprintf( "disk(%s) already attached to node(%s), could not be attached to node(%s)", diskURI, *disk.ManagedBy, nodeName) attachedNode := path.Base(*disk.ManagedBy) klog.V(2).Infof("found dangling volume %s attached to node %s", diskURI, attachedNode) danglingErr := volerr.NewDanglingError(attachErr, types.NodeName(attachedNode), "") return -1, danglingErr } } </ins> vmset, err := c.getNodeVMSet(nodeName) if err != nil { return -1, err

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Pods with PVs could not be moved from one node to another. According to the controller-manager logs, K8s tried to detach to Azure disks from the same node. One of the detach operation waited for this lock (). While it waited for the lock, the controller-manager pod was restarted (for whatever reason, not related to this bug, lets assume an operator restarted the pod). After the restart, K8s only tried to attach the disk to the new node but is not trying to detach it first. What you expected to happen: After a restart of the controller-manager, it should continue unfinished detach operations How to reproduce it (as minimally and precisely as possible): 2 Pods with PVs running on one node Move both pods to another node restart the controller-manager shortly after the move Anything else we need to know?: Environment: Kubernetes version (use ): 1.15.1 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig azure
/sig cloud-provider
/area provider/azure that's expected behavior, k8s controller won't continue detach volume if controller manager restarted, user needs to detach that disk volume manually.
Could you please elaborate why this behaviour is expected. I would assume that the controller manager tries to reach the desired state after a restart? Then it would need to detect that the volume is still attached to the wrong node and should try to detach it once more, before it continuing to attach the (still attached) volume? This leads to many ... Why is it required to resolve this situation by manual interaction?
I can't imagine that "manual intervention" is "desired behaviour" (yes, you wrote "expected", but that doesn't matter, if you make no proposal, how to improve this). If this isn't fixed, we can't use Azure, period. We can't have clusters that need to be babysitted / you cannot expect people running workloads on Azure clusters and babysit them and manually fixing what should/can fix itself.

Level-triggered behaviour and resilience is what made Kubernetes so successful... please bring that also to Azure's KCM/CCM.
do you know why controller manager is restarted, it's not a common scenario, we cannot guarantee disk attach/detach functionality works if controller manager restarted at that time.
related to
I think there can be many reasons why the controller-manager is restarted during running detach or attach operations e. g. the hosting machine turn into error state, if you run the controller-manager in k8s -vertical pod autoscaling,... etc. I understand you can't guarantee that attach/detach works when the controller-manager is restarted, but I don't understand why the controller-manager can't recover the situation autonomously after a restart. After restart it tries to attach the volume of the rescheduled pod and this does not work because the volume is still attached to another machine. This is indicated by a error from the arm and the controller-manager could react to it properly, right?
This is an upstream issue, and there is a discussion in , let's check whether we can fix it in cloud provider driver in that thread. Thanks.
I will try to fix this issue in azure disk driver, thanks all for the discussion. details: /assign
Thank you very much It means a lot to us, that there are no manual operations. I wonder how you guys deal with it in AKS, since Azure uses the same controllers, right? Anyway, it helps tremendously, that the clusters are self-healing.
AKS use same kube-controller-manager, while it won't be restarted easily.
OK, but how? I understand that AKS only supports 100 nodes. So, I guess you have some upper limit and just hope, nobody stresses it, because if this happens, you it either restarts or you need to scale it up and then it gets restarted as well. Also, regular maintenance, low-level as well as Kubernetes updates, all must impact it. Considering the probably thousands over thousands of clusters (I don't know, but I expect you run many), any such problem, even if unlikely by itself, must be in your interest to fix, simply because the number of clusters increases the likelihood that it happens, I would think. Anyway, thank you for making this change
Yes, that's reasonable, I will try to fix it as my top priority.


Question: match: configs[DistrolessIptables] = Config{list.BuildImageRegistry, "distroless-iptables", "v([0-9]+).([0-9]+).([0-9]+)"} - name: "registry.k8s.io/go-runner: dependents" <del> version: v2.3.1-go1.23rc2-bookworm.0 </del> <ins> version: v2.3.1-go1.22.5-bookworm.0 </ins> refPaths: - path: build/common.sh match: __default_go_runner_version=

sgtm, add an issue labeled as release-blocking and in the milestone to either promote to 1.23.0 or revert this to 1.22.x before our 1.31 release Originally posted by in Kubernetes is being built and tested with go1.23rc2 at HEAD () to let it bake in the CI and help iron out issues, if any, with this version of Go. However, for the 1.31 release, the version of Go should be a non-rc, released version. The release timeline of go1.23.0 is ~same as the release timeline of K8s 1.31.0 (see ). If go1.23.0 is not out by then, we should revert to go1.22.x before cutting K8s 1.31.0. /sig release architecture /area code-organization /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted /cc
/milestone v1.31
cc
xposting relevant slack thread:
Also unlike in the past, missing shipping 1.23.0 in 1.31.0 is not a big deal, we have much more room to switch to 1.23.0 in a patch release after all the compat changes with the go team. So we should not fear reverting if the timelines don't line up.
(though go1.23 adds the thing that makes it workable to default to old go version compat settings, so we'll have to eke out another 5 months on 1.21/1.22 for until we can adopt 1.23 on release branches in ~Jan if we miss 1.31, but that's how it looks like it will have to happen)


Question: allErrs = append(allErrs, field.Invalid(fldPath.Child("targetPort"), sp.TargetPort, PortNameErrorMsg)) } <del> if isHeadlessService { if sp.TargetPort.Type == intstr.String || (sp.TargetPort.Type == intstr.Int && sp.Port != sp.TargetPort.IntValue()) { allErrs = append(allErrs, field.Invalid(fldPath.Child("port"), sp.Port, "must be equal to targetPort when clusterIP = None")) } } </del> <ins> // in the v1 API, targetPorts on headless services were tolerated. // once we have version-specific validation, we can reject this on newer API versions, but until then, we have to tolerate it for compatibility. // // if isHeadlessService { // 	if sp.TargetPort.Type == intstr.String || (sp.TargetPort.Type == intstr.Int && sp.Port != sp.TargetPort.IntValue()) { // 		allErrs = append(allErrs, field.Invalid(fldPath.Child("targetPort"), sp.TargetPort, "must be equal to the value of 'port' when clusterIP = None")) // 	} // } </ins> return allErrs }

In 1.0 and 1.1, headless services that specified a mismatched port/targetPort were tolerated (the targetPort was ignored). tightened validation to reject these services. That causes problems when upgrading from currently-working clusters.
2 candidate to avoid breaking backwards compatibility
oh hell. Yeah. I don't think this is worth a breaking change. We should either roll that back ASAP or change it to nuke any user-provided value and replace it with the value. I think the latter is potentially dangerous and also not worth the risk, so I think we should rollback and just update the docs for targetPort. objections?
Who has time to do this? It doesn't look like an auto-rollback is doable.
Already opened a PR in
Yes, I agree this isn't worth a breaking change, But could we give an warning log that targetPort isn't equal to port and would be ignored?
There's no facility in validation for warning. I don't think spamming the server log in response to API requests is that helpful here
Yes, that sounds reasonable. I just think that targetPort was ignored is a little unfriendly to user.


Question: portStr = append(portStr, fmt.Sprintf("%s/%d", p.Protocol, p.Port)) } <del> affinityType := apiService.Spec.SessionAffinity </del> serviceName := types.NamespacedName{Namespace: apiService.Namespace, Name: apiService.Name} lbRefStr := fmt.Sprintf("%v(%v)", loadBalancerName, serviceName) glog.V(2).Infof("ensureExternalLoadBalancer(%s, %v, %v, %v, %v, %v)", lbRefStr, gce.region, requestedIP, portStr, hostNames, apiService.Annotations)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area cloudprovider What happened: The cloud provider interface that should . The GCE provider does this for but not for . Every other in-tree provider implements by fully updating the load balancer including updated backend nodes: , , , And the following out-of-tree providers do as well and Oracle (not oss yet). What you expected to happen: I expect that all cloud providers implement by creating or fully updating the load balancer if it already exists. Related discussion: /cc
/assign Make sense to make implementation in GCE consistent with others for the ease of up-coming service controller enhancement. This sounds like a 1.10 task.
is this task blocking you on another attempt at refactoring the service controller sync logic?
yup


Question: import ( "fmt" "os" <ins> "strings" </ins> "github.com/golang/glog" "golang.org/x/exp/inotify"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Repport Kubernetes version (use ): 1.5.1 (Both Server and Client) Environment: Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): UBUNTU 16.04 Kernel (e.g. ): 4.4.0-47-generic Install tools: Chef What happened: Kubelet picks up dot files in --pod-manifest-path causing stuff to breaks by replacing existing pods. What you expected to happen: Exclude dot files from being picked up under --pod-manifest-path How to reproduce it (as minimally and precisely as possible): Take any manifest, copy it as and kubelet will deploy that new manifest Anything else do we need to know: From the slack channel it's been identified to be introduce by (by on Slack)
Just sent a simple fix, while still open for discussion.


Question: } forget := false <ins> // Check if the number of jobs succeeded increased since the last check. If yes "forget" should be true // This logic is linked to the issue: https://github.com/kubernetes/kubernetes/issues/56853 that aims to // improve the Job backoff policy when parallelism > 1 and few Jobs failed but others succeed. // In this case, we should clear the backoff delay. if job.Status.Succeeded < succeeded { forget = true } </ins> // no need to update the job if the status hasn't changed since last time if job.Status.Active != active || job.Status.Succeeded != succeeded || job.Status.Failed != failed || len(job.Status.Conditions) != conditions { job.Status.Active = active

This issue was born out of the discussion from and In short: Job controller has an exponential backoff in place when failures happen to a job. As long as the parallelism is one, it's clear when a failure or success happens, the latter clears the backoff. The problem is when parallelism 1, which can lead to situation where the controller can observe successful and failed pod at the same time and won't clear the backoff leading to further retries and longer wait times (see for extensive details about the problem in e2e) Possible solutions: everything as is both success and failure are observed clear the backoff Pros and cons: Ad. 1. Aside from () I wonder if users got affected by it. Does it make sense to fix it at all? Ad. 2. This approach make sense in some situation, but the question arise, should that be controlled by a user. What if I have 9 failures and 1 success (parallelism = 10), should I still clear the backoff? Maybe we should introduce user controller field to set the failure rate in this case.
/cc
(jumping here 'cause forwarded me this). I personally think that the right approach is to reset the back-off as soon we have a successful pod run to complete. Back-off was introduced, at least in our case, to reduce the impact of bad configured PodSpec in Jobs. If in your run you do have some success, it means you've a good config and you just keep trying (not that different that what we did before ). I don't think it's worth asking user finer tune config. is already complex enough. My 2 cents.
cc
I agree with any success should immediately clear the backoff flag, since it shows the job is configured properly and the problem is only temporary. I'll assign it to myself, but anyone is open to submit a PR fixing this.
Hi I started to implement the pod success check in order to clear the backoff delay in case of at least one of the pods succeed. let me know if you are fine with this approach, and then I will go further with unit-tests and end2end tests.
is the rest of the work on this issue getting done for 1.10? Please update, and add approved-for-milestone if this is on track for 1.10.
I updated my change with missing unit-tests and created the PR:


Question: }, } <ins> topologyDupValues := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1", "node1"}, }, }, }, } topologyMultiValues := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1", "node2"}, }, }, }, } topologyEmptyMatchLabelExpressions := []api.TopologySelectorTerm{ { MatchLabelExpressions: nil, }, } topologyDupKeys := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node2"}, }, }, }, } topologyMultiTerm := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node2"}, }, }, }, } topologyDupTermsIdentical := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, } topologyExprsOneSameOneDiff := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node2"}, }, }, }, } topologyValuesOneSameOneDiff := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1", "node2"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1", "node3"}, }, }, }, } topologyDupTermsDiffExprOrder := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1"}, }, { Key:    "kubernetes.io/hostname", Values: []string{"node1"}, }, }, }, } topologyDupTermsDiffValueOrder := []api.TopologySelectorTerm{ { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone1", "zone2"}, }, }, }, { MatchLabelExpressions: []api.TopologySelectorLabelRequirement{ { Key:    "failure-domain.beta.kubernetes.io/zone", Values: []string{"zone2", "zone1"}, }, }, }, } </ins> cases := map[string]bindingTest{ "no topology": { class:         makeClass(nil, nil),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!


Question: } } <del> func TestGetPodStatusFromNetworkPlugin(t *testing.T) { cases := []struct { pod                *v1.Pod fakePodIP          string containerID        string infraContainerID   string networkStatusError error expectRunning      bool expectUnknown      bool }{ { pod: &v1.Pod{ ObjectMeta: v1.ObjectMeta{ UID:       "12345678", Name:      "foo", Namespace: "new", }, Spec: v1.PodSpec{ Containers: []v1.Container{{Name: "container"}}, }, }, fakePodIP:          "10.10.10.10", containerID:        "123", infraContainerID:   "9876", networkStatusError: nil, expectRunning:      true, expectUnknown:      false, }, { pod: &v1.Pod{ ObjectMeta: v1.ObjectMeta{ UID:       "12345678", Name:      "foo", Namespace: "new", }, Spec: v1.PodSpec{ Containers: []v1.Container{{Name: "container"}}, }, }, fakePodIP:          "", containerID:        "123", infraContainerID:   "9876", networkStatusError: fmt.Errorf("CNI plugin error"), expectRunning:      false, expectUnknown:      true, }, } for _, test := range cases { dm, fakeDocker := newTestDockerManager() ctrl := gomock.NewController(t) fnp := mock_network.NewMockNetworkPlugin(ctrl) dm.networkPlugin = fnp fakeDocker.SetFakeRunningContainers([]*FakeContainer{ { ID:      test.containerID, Name:    fmt.Sprintf("/k8s_container_%s_%s_%s_42", test.pod.Name, test.pod.Namespace, test.pod.UID), Running: true, }, { ID:      test.infraContainerID, Name:    fmt.Sprintf("/k8s_POD.%s_%s_%s_%s_42", strconv.FormatUint(generatePodInfraContainerHash(test.pod), 16), test.pod.Name, test.pod.Namespace, test.pod.UID), Running: true, }, }) fnp.EXPECT().Name().Return("someNetworkPlugin").AnyTimes() var podNetworkStatus *network.PodNetworkStatus if test.fakePodIP != "" { podNetworkStatus = &network.PodNetworkStatus{IP: net.ParseIP(test.fakePodIP)} } fnp.EXPECT().GetPodNetworkStatus(test.pod.Namespace, test.pod.Name, kubecontainer.DockerID(test.infraContainerID).ContainerID()).Return(podNetworkStatus, test.networkStatusError) podStatus, err := dm.GetPodStatus(test.pod.UID, test.pod.Name, test.pod.Namespace) if err != nil { t.Fatal(err) } if podStatus.IP != test.fakePodIP { t.Errorf("Got wrong ip, expected %v, got %v", test.fakePodIP, podStatus.IP) } expectedStatesCount := 0 var expectedState kubecontainer.ContainerState if test.expectRunning { expectedState = kubecontainer.ContainerStateRunning } else if test.expectUnknown { expectedState = kubecontainer.ContainerStateUnknown } else { t.Errorf("Some state has to be expected") } for _, containerStatus := range podStatus.ContainerStatuses { if containerStatus.State == expectedState { expectedStatesCount++ } } if expectedStatesCount < 1 { t.Errorf("Invalid count of containers with expected state") } } } </del> func TestSyncPodGetsPodIPFromNetworkPlugin(t *testing.T) { const ( containerID      = "123"

go test -v This is because there are a few test cases in dockertools are linux-specific.


Question: - [Scheduler pod template](#scheduler-pod-template) - [Controller Manager Template](#controller-manager-template) - [Starting and Verifying Apiserver, Scheduler, and Controller Manager](#starting-and-verifying-apiserver-scheduler-and-controller-manager) <del> - [Logging](#logging) - [Monitoring](#monitoring) - [DNS](#dns) </del> <ins> - [Starting Cluster Services](#starting-cluster-services) </ins> - [Troubleshooting](#troubleshooting) - [Running validate-cluster](#running-validate-cluster) - [Inspect pods and services](#inspect-pods-and-services)

has TODOs for how to bring up cluster addons like monitoring and logging. We should fill these in. requested by user:


Question: } }) <del> By("not listing pulled images [Conformance]", func() { </del> <ins> By("not listing pulled images", func() { </ins> image, _ := NewConformanceImage("docker", "") tags, err := image.List() Expect(err).ShouldNot(HaveOccurred())

As of now node e2e requires all the tests to pass. There are tests like the container conformance tests which have been flaky in the past - , , We need ability to mark certain tests as flaky and not run them as part of merge queue.
Ginkgo flags makes it possible to skip tests. We can use a prefix to not run flaky tests in our existing jenkins CIs.
Closing via


Question: p:             &ExecOptions{StreamOptions: StreamOptions{PodName: "foo"}}, args:          []string{"cmd"}, argsLenAtDash: -1, <del> expectedPod:   "foo", </del> <ins> expectedPod:   execPod(), </ins> expectedArgs:  []string{"cmd"}, name:          "pod in flags", }, { <ins> p:             &ExecOptions{StreamOptions: StreamOptions{PodName: "pod/foo"}}, args:          []string{"cmd"}, argsLenAtDash: -1, expectedPod:   execPod(), expectedArgs:  []string{"cmd"}, name:          "pod with 'pod/' prefix in flags", }, { </ins> p:             &ExecOptions{}, args:          []string{"foo", "cmd"}, argsLenAtDash: 0,

I feel like I should be able to do: but I get
or we have to do something like this: sounds reasonable to me. needs more comments.
All commands that accept resources should accept resource/NAME syntax. In this case for back compat we must only accept a single argument, is familiar with this.
Exactly. ResourceNames in the resource builder is what is needed here to handle the default resource (pod in this case) and provide the ability to specify more.
Related
If nobody is working on it, I'd like to pick the issue, pls.
I don't think anyone is. , Illia Khudoshyn wrote:
So could you please assign me?
I can't assign to just anyone, unfortunately, but by responding here, anyone else who might work on it will see that you are already in progress. , Illia Khudoshyn wrote:
please go ahead, you did own this now. :)
Has this been reverted again? I cannot use “pod/podname” for v1.11:
^ Seconding the above. I'm getting the same 'may not contain /' error. Naturally it's trivial to use unix commands to filter this, but this is such a basic feature and matches the way the rest of kubectl works, so it's quite a shocking omission.
as a unix novice who's also experiencing this problem, I'd love to know your workaround
If you use selectors, you can do something like this: Otherwise you'll need to manually grep, something like this: is the syntax in bash to run the command and use its output as part of another command. You can also use backticks to achieve the same effect but that's considered bad form by many (myself included). For example: If going the grep route, take care to make sure your regex will only return one result, otherwise you'll have to pipe it to xargs or something similar.
Just ran into this on a brand new default installation. Local Mac Docker for Desktop Enabled k8 in preferences error: invalid resource name "svc/broken-gorilla-postgresql": [may not contain '/']
FWIW you can use combined with the flag, e.g.
Had the same problem. Updating kubectl resolved the issue for me.


Question: "//pkg/apis/core/v1/helper:go_default_library", "//pkg/cloudprovider:go_default_library", "//pkg/controller:go_default_library", <ins> "//pkg/kubelet/apis:go_default_library", </ins> "//pkg/util/mount:go_default_library", "//pkg/volume:go_default_library", "//pkg/volume/util:go_default_library",

Resizing a persistent volume is partially broken on openstack (cinder) with k8s v1.11.1. Partially in the sense that static resizing (requiring a pod restart) works, but the newly introduced online resizing (not requiring a pod restart) fails. Upon resizing a pvc it remains in the Resizing condition forever (see the warning event) until I delete the pod that has this pvc mounted: After pod deletion it notifies me (correctly) that it's now waiting for the pod to come up to finish resizing: So far the warning event does not break the ExpandPersistentVolumes feature, as this one anyhow requires a pod restart. However it breaks the newly introduced ExpandInUsePersistentVolumes feature (alpha with k8s v1.11; doesn't require a pod restart for resizing to finish), as the pvc will remain in "Resizing" condition forever (or until the pod is restarted but this defeats the purpose of ExpandInUsePersistentVolumes). I'm not sure whether this is fully related but upon digging around I found and a consecutive fix with which was merged on Jan 23. However a few days later this fix seems to be (probably accidentally) overwritten with this commit: /kind bug /sig storage /sig openstack
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
It is already fixed. So I will close it.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I am trying cinder online pvc resizing on kubernetes1.12.2. I noticed this kubernetes version has include your fix, but I still failed to resize pvc online.

The error log has no change: Warning VolumeResizeFailed 45s (x2 over 66s) volumeexpand Error expanding volume "default/myclaim-expend" of plugin : volume status is not available The error log is reported below code line "volume.Status != volumeAvailableStatus" // ExpandVolume expands the size of specific cinder volume (in GiB) func (os *OpenStack) ExpandVolume(volumeID string, oldSize resource.Quantity, newSize resource.Quantity) (resource.Quantity, error) { volume, err := os.getVolume(volumeID) if err != nil { return oldSize, err } if volume.Status != volumeAvailableStatus { // cinder volume can not be expanded if its status is not available return oldSize, fmt.Errorf("volume status is not available") } One point confused me, once cinder volume is used by pod, it's status is "in-use", how the online pvc resizing pass the code check "volume.Status != volumeAvailableStatus"? There's where the online pvc resizing fails.
It looks like Cinder can not support online volume expansion - which is what is happening here. Even if online expansion is enabled, I think to actually perform resizing of Cinder volumes - you must bring the pod down.


Question: }, }, }, <ins> "metadata.name: must be no more than 52 characters": { ObjectMeta: metav1.ObjectMeta{ Name:      "10000000002000000000300000000040000000005000000000123", Namespace: metav1.NamespaceDefault, UID:       types.UID("1a2b3c"), }, Spec: batch.CronJobSpec{ Schedule:          "* * * * ?", ConcurrencyPolicy: batch.AllowConcurrent, JobTemplate: batch.JobTemplateSpec{ Spec: batch.JobSpec{ Template: validPodTemplateSpec, }, }, }, }, </ins> "spec.jobTemplate.spec.manualSelector: Unsupported value": { ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: We have some cron jobs with longish names (~60 characters), and noticed that the corresponding jobs weren't being created by the cronjob controller because the name was too long. Here's what happened: There's a 63-character limit on job names (because for some reason on job creation the name gets copied into a label, where the limit is 63 chars). For example, if I try to create a job with a long name, I get the error message: By itself, this is fine. I can just make a job with a shorter name. However! If I create a cron job, the cron job controller will create jobs named . So the job's name is 11 characters longer than the job name. This means that if I create a cron job with a 63-character name, it'll result in a job name that's 74 characters, which the API will refuse to accept. This means the cron job never gets scheduled. Some possible resolutions: validation on cron job creation that refuses to let you create jobs with names 52 characters the cron job controller truncate cronjobs names remove the 63-character limit on job names (this seems difficult) There's actually a deeper problem here at play: Kubernetes' resource name limit is 253 characters. But the limit for resouce labels is 63 characters. So when a job is created, the name (which is allowed to be up to 255 chars) gets copied into a label, which has a lower limit. So the job creation fails. (these are the ) What you expected to happen: When I create a cron job, I expect it to either to create the cron job, or jobs correctly from that cron job How to reproduce it (as minimally and precisely as possible): a cronjob with a name that's more than 52 characters that it does not schedule jobs, with an error message like Anything else we need to know?: Kubernetes version: 1.7.3 Cloud provider: AWS OS: Ubuntu 16.04 Kernel: 4.4.0 example of a job definition with a long name This job definition is invalid:
There are no sig labels on this issue.

Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps
cc


Question: <ins> # See the OWNERS docs at https://go.k8s.io/owners # This is the owner of the test code.  The test data itself is owned by sig-architecture. reviewers: - aojea - cheftako - oomichi - johnbelamaric approvers: - cheftako - spiffxp - johnbelamaric emeritus_approvers: labels: - area/conformance - sig/architecture </ins>

I can create a cluster with one node and pass conformance, ~there are some tests that require at least 2 nodes to run~ ~~ ~This is because the tests Skip if they are not able to run~ ~~ ~As consequence, the list of tests for conformance depends on the number of nodes, a single cluster node (that is not a cluster by definition of cluster :)) will run less tests than a multi cluster node and still be Conformant~ EDIT I have to correct myself, it seems that the conformance tests doesn't skip, the problem is: there are no conformance tests that exercise anything that requires more than one node, hence, single node cluster can pass conformance. I expect a cluster (a group of similar things or people positioned or occurring closely together) and a "distributed system" to not be able to be single-node. In addition, lots of most important Kubernetes features can not be exercised on one node, lowering the quality bar. local- pass conformance Related discussion <details</details<details</details<details</details<details</details<details</details<details</details
/assign Maybe this has changed, but last time I've checked the minimum for conformance was 1 control-plane and 2 workers, if that didn't change I will submit a PR to enforce it in the suite Regarding sig-network, with one node you can not verify the networking model, where pod to pod in different nodes should be able to communicate without NAT, that is a strong requirement ... I have to review closely what tests do we have in Conformance
/area conformance /sig network
that seems pretty intentional... changing the definition of conformance to require multiple nodes probably deserves more feedback than an issue/PR
I'm not changing the definition, just asserting it
if we are testing features like replicaset, daemonset, services, ..., in one node only we are missing a lot of testing on those features ... a podman/docker single node can be a kubernetes conformance cluster ... do I need a scheduler or network for anything, I'm just deploying always in the same node?
No I don't think so. Where are your results and configuration? You need two schedulable nodes to pass the conformance suite. That's not correct, we test rolling pod behavior with replicasets. This test requires multiple nodes. There is at least one.

That job is not passing conformance, from the configuration: (see: ) Conformance requires not setting --, you must run all tests without skipping. We don't, at the very least replicaset is being tested multi-node, just not in this job that is not running the full conformance suite. When conformance results are submitted they are now validated to ensure that all tests were run.


Question: //   - If a file is found, then retreives volumePluginDependentPath from globalMapPathUUID. //   - Once volumePluginDependentPath is obtained, store volume information to VolumeSource // examples: <del> //   mapPath: pods/{podUid}}/{DefaultKubeletVolumeDevicesDirName}/{escapeQualifiedPluginName}/{volumeName} </del> <ins> //   mapPath: pods/{podUid}/{DefaultKubeletVolumeDevicesDirName}/{escapeQualifiedPluginName}/{volumeName} </ins> //   globalMapPathUUID : plugins/kubernetes.io/{PluginName}/{DefaultKubeletVolumeDevicesDirName}/{volumePluginDependentPath}/{pod uuid} func (plugin *fcPlugin) ConstructBlockVolumeSpec(podUID types.UID, volumeName, mapPath string) (*volume.Spec, error) { pluginDir := plugin.host.GetVolumeDevicePluginDir(fcPluginName)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently, FC plugin returns volume name and empty FCVolumeSource during ConstrutVolumeSpec during . However, in order to recreate newMounter from volumeSpec, volume information are lacking. As a result, filesystem volume's reconstruction always fails. What you expected to happen: Filesystem volume's reconstruction works properly, then a volume is unmounted and detached from kubelet node. How to reproduce it (as minimally and precisely as possible): (1) Create filesystem volume using FC plugin (2) Restart kubelet service which a pod with FC volume is running Anything else we need to know?: cc Environment: Kubernetes version (use ):Install tools: local-cluster-
/assign
/sig storage


Question: } if cstatus.State.Terminated != nil { <del> // Since the container is terminated, we know it is safe to // remove it without any reconciliation. Removing the container // will also remove it from the `containerMap` so that this // container will be skipped next time around the loop. </del> <ins> // The container is terminated but we can't call m.RemoveContainer() // here because it could remove the allocated cpuset for the container // which may be in the process of being restarted.  That would result // in the container losing any exclusively-allocated CPUs that it // was allocated. </ins> _, _, err := m.containerMap.GetContainerRef(containerID) if err == nil { <del> klog.Warningf("[cpumanager] reconcileState: skipping container; already terminated (pod: %s, container id: %s)", pod.Name, containerID) err := m.RemoveContainer(containerID) if err != nil { klog.Errorf("[cpumanager] reconcileState: failed to remove container (pod: %s, container id: %s, error: %v)", pod.Name, containerID, err) failure = append(failure, reconciledContainer{pod.Name, container.Name, containerID}) } </del> <ins> klog.Warningf("[cpumanager] reconcileState: ignoring terminated container (pod: %s, container id: %s)", pod.Name, containerID) </ins> } continue }

What happened: A container which had exclusive CPUs assigned to it (and an entry in /var/lib/kubelet/cpumanagerstate specifying those CPUs) was killed. When it came back, it was using the default cpuset, and its entry in /var/lib/kubelet/cpumanagerstate was gone. What you expected to happen: It should have been given exclusive CPUs. How to reproduce it (as minimally and precisely as possible): Create a pod/container with "guaranteed" QoS. Determine the exclusive CPUs allocated to the container. From the host, kill pid1 within the container. ("kill -9" works.) The container will restart within the existing pod, but it will be allocated cpus from the default cpuset (which will now include the CPUs that had been exclusively allocated to the container earlier). Anything else we need to know?: I some instrumentation to kubelet. What seems to be happening is this: 1) The container is killed (or dies). Kubelet then immediately creates a new container and calls PolicyStatic.Allocate() to allocate exclusive CPUs for it. This updates the entry in /var/lib/kubelet/cpumanagerstate (and the in-memory cache) which is mapped by podUID and container name, such that it contains the cpuset for the new container. 2) The InternalContainerLifecycle.PostStopContainer() code runs, which calls i.cpuManager.RemoveContainer(containerID). This uses the containerID of the exited container to look up the podUID/container-name tuple, then deletes the cpuset entry from /var/lib/kubelet/cpumanagerstate (and the in-memory cache). But the entry that was deleted was actually the one for the new container. 3) Kubelet calls the CPU manager's AddContainer() function, which calls m.state.GetCPUSetOrDefault(string(p.UID), c.Name) to get the cpuset based on the podUID and container name of the new container. The cpuset entry was erroneously deleted, so this returns the default cpuset, and the container which should have gotten exclusive CPUs ends up running on the default cpuset. An solution suggested by is to delay calling cpuManager.RemoveContainer() until the whole pod is deleted. This would preserve the CPU allocations.

An alternate solution also suggested by is to avoid calling cpuManager.RemoveContainer() from the container lifecycle code, but instead call removeStaleState() in the CpuManager.Allocate() function. Environment: Kubernetes version (use ): 1.18.1 Cloud provider or hardware configuration: x86, SMP OS (e.g: ): CentOS 7 Kernel (e.g. ): 3.10.0-1062.1.2.rt56.1025.el7
/sig sig-node
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig sig-node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Considering the following change based on discussion over Slack with
Sorry that bother you via this close issue, but I was tracing back to this issue via the PR. As per your statement above, if I understand correctly, this problem only happened when the container inside Pod got killed. but an simple Pod restarted (e.g. delete/re-create via kubectl delete pod command) should not trigger the problem, right? Or any container(inside Pod) or Pod restart with guaranteed Qos will trigger this issue?


Question: } } <ins> func TestKindForWithNewCRDs(t *testing.T) { tests := map[string]struct { in       schema.GroupVersionResource expected schema.GroupVersionKind srvRes   []*metav1.APIResourceList }{ "": { in:       schema.GroupVersionResource{Group: "a", Version: "", Resource: "sc"}, expected: schema.GroupVersionKind{Group: "a", Version: "v1", Kind: "StorageClass"}, srvRes: []*metav1.APIResourceList{ { GroupVersion: "a/v1", APIResources: []metav1.APIResource{ { Name:       "storageclasses", ShortNames: []string{"sc"}, Kind:       "StorageClass", }, }, }, }, }, } for name, test := range tests { t.Run(name, func(t *testing.T) { invalidateCalled := false fakeDiscovery := &fakeDiscoveryClient{} fakeDiscovery.serverResourcesHandler = func() ([]*metav1.APIResourceList, error) { if invalidateCalled { return test.srvRes, nil } return []*metav1.APIResourceList{}, nil } fakeCachedDiscovery := &fakeCachedDiscoveryClient{DiscoveryInterface: fakeDiscovery} fakeCachedDiscovery.invalidateHandler = func() { invalidateCalled = true } fakeCachedDiscovery.freshHandler = func() bool { return invalidateCalled } // in real world the discovery client is fronted with a cache which // will answer the initial request, only failure to match will trigger // the cache invalidation and live discovery call delegate := NewDeferredDiscoveryRESTMapper(fakeCachedDiscovery) mapper := NewShortcutExpander(delegate, fakeCachedDiscovery) gvk, err := mapper.KindFor(test.in) if err != nil { t.Errorf("unexpected error: %v", err) } if diff := cmp.Equal(gvk, test.expected); !diff { t.Errorf("unexpected data returned %#v, expected %#v", gvk, test.expected) } }) } } </ins> type fakeRESTMapper struct { kindForInput schema.GroupVersionResource }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I created a crd e.g.: Then I used . I found that the first time the result will be with an error: And as for the second time it will be normal: And it can be stably reproduced every time I created a new CRD. What you expected to happen: kubectl returns the first time I requested the new CRD instead of an error. Anything else we need to know?: I also found that requesting short name e.g. will be the same. It returns an error the first time but becomes normal the second time. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug
/area custom-resources
Confirmed that could show up even the crd has status /assign
/sig api-machinery Haven't looked into the problem yet but could be related:
Looks like this happens because cache file in .kube/http-cache/ or .kube/cache/ is not synced with kube-apiserver. delete all files under .kube/http-cache/ and .kube/cache/ will cause a sync and the error will not show up. This looks like a cli issue.
fyi
Tried to fix in cli:
/cc
/assign
this seems similar to
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is /shrug
Can you put this into words? What's the actual issue?
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is Can you put this into words? What's the actual issue? SRY, I should have put it clearer.. I just mean that this issue is sort of fundamental and obviously has been existing for a long time.
/remove-shrug
/unshrug looks like an important problem we want to solve (?)
¯()/¯ <detailsIn response to : /unshrug Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten ouch..
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
encountered same issue, as suggested the error will not show up.
any progress?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a little more info, and here is how I am able to reproduce it (kubectl version 1.18.0 and cluster version 1.19.0-beta.2): A couple interesting things i noted... If I deleted my and re-ran it, I could not reproduce it. It is only the first time the short name was EVER used that this occurs. I only tried from one client machine, so I'm not sure this is cluster-specific or client-specific. If I used a different short name, like 'fo2' and re-ran the example, then it reproduces the problem. You can also run after creating the CRD and it has the same effect as deleting your . EDIT TO ADD: This doesn't seem to be a timing thing... if I sleep for 10 seconds after creating the CRD, it still has the problem.
The root of the problem is that the is run before the discovery. So in this particular case this line continues to b/c read cached data and never reached out to server. Whereas discovery mapper in will get a not found error and will proceed invalidating cache and re-fetching discovery data and return that back. In the subsequent call the shortcut expander is already aware of the new shortcut so it works just fine.

We'd need to teach shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.
See for possible approach.
Any new progress on this ?
lists a candidate solution. This bug is causing steadily failing tests. I'll talk to a few people to see if this can get any traction. do you know who can get this moving? It sounds like you have identified the problem and have proposed a solution.


Question: BirthCry() StartGarbageCollection() ListenAndServe(kubeCfg *kubeletconfiginternal.KubeletConfiguration, tlsOptions *server.TLSOptions, auth server.AuthInterface, tp trace.TracerProvider) <del> ListenAndServeReadOnly(address net.IP, port uint) </del> <ins> ListenAndServeReadOnly(address net.IP, port uint, tp trace.TracerProvider) </ins> ListenAndServePodResources() Run(<-chan kubetypes.PodUpdate) RunOnce(<-chan kubetypes.PodUpdate) ([]RunPodResult, error)

Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first.

Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in


Question: } updated := accessor.GetInitializers() <ins> // controllers deployed with an empty initializers.pending have their initializers set to nil // but should be able to update without changing their manifest if updated != nil && len(updated.Pending) == 0 && updated.Result == nil { accessor.SetInitializers(nil) updated = nil } </ins> existingAccessor, err := meta.Accessor(a.GetOldObject()) if err != nil { // if the old object does not have an accessor, but the new one does, error out

(SEE UPDATE at ) Currently, if you're writing an initializer controller, you're most likely using it to initialize a and deploying the controller with . Therefore you're likely using the following trick to avoid the chicken and egg problem: This bypasses the initializers for the pods and lets your controller to be deployed even though your InitializerConfiguration is configured to capture all pods. Problem occurs when you apply an update to this Deployment with the same manifest containing empty pending list. You get error: I think this is a somewhat major UX issue. It requires users to deploy the initializer controller with the but later on when they need to update the controller with , they can't use the same manifest containing the empty list. /area api /area extensibility /sig api-machinery /kind bug /assign
/cc
We silently ignore some metadata updates, which is one option. Potentially we shouldn't require empty initializers pending, but simply empty initializers (which means - init directly), and only ignore that. Are there any examples of use of SMP for similar? I'm not sure this meets the bar for 1.8 but if we can find an example of something consistent already in the API I'd be willing to review.
I don't have any concrete examples, there's an example initializer at making use the trick, I use it myself, and I suspect many more initializers to come will use this, too. I'm not sure if this would work, but does setting provide a way to skip initialization (if possible at all)? If not, we might actually need a separate field to signal skipping initialization.
/cc
Currently the initializer plugin to on UPDATE, but that happens after it updating already initialized objects. We can simply change their order.
I'd like to amend the issue, it's more serious than I thought. (thanks for bringing it to my attention If you're deploying your Initializer with and if you have an initializerconfiguration for objects, it's impossible to deploy your initializer in the first place without deleting InitializerConfiguration first: is not inherited by the created on is not honored (this field is not propagated to the created Pod). I think (2) should be fixed somehow, otherwise there will be no way of writing an initializer for Pods and deploying the initializer itself using Deployment/ReplicaSet/RC without deleting the InitializerConfiguration first.

Maybe what we need is an exclusion/exemption selector (or namespace(s)) as discussed in to allow deployment of Initializers themselves easier.
For 1, see I don't want to pursue that though, because i) we'll need to fix workload controllers one by one, and ii) we need to carefully consider permission issues. I think we should focus on adding the namespace selector.


Question: // newExecPodSpec returns the pod spec of exec pod func newExecPodSpec(ns, generateName string) *v1.Pod { <del> pod := NewAgnhostPod(ns, "agnhost-pod", nil, nil, nil) </del> <ins> // GenerateName is an optional prefix, used by the server, // to generate a unique name ONLY IF the Name field has not been provided pod := NewAgnhostPod(ns, "", nil, nil, nil) </ins> pod.ObjectMeta.GenerateName = generateName return pod }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are failing: [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] Since when has it been failing: Constant failure from 2020-09-23 Testgrid link: Reason for failure: From Similar logs found in previous runs Anything else we need to know: There is open bug () for flakes, however this is not a flake, but a constant failure. /sig networking
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Which jobs are failing: ci-kubernetes-e2e-gce-scale-correctness [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] Constant failure from 2020-09-23 From Similar logs found in previous runs There is open bug () for flakes, however this is not a flake, but a constant failure. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/priority critical
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /priority critical Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent
ok, I think I got it,
so, it seems the 500 error is generated because it is using GenerateName, but ti has the pod.Name hardcoded
So do we know if this is related to any single proxy mode or applies for all?

Trying to figure out how it relates to & since those were only pertaining to IPVS.
This is a new problem that affected the same test in all modes, it was not a network problem at the end The ipvs one was flaky before this, I'll reopen it


Question: t.Errorf("error while waiting for pods in namespace %v: %v", ns, err) } } <ins> func waitForSchedulerCacheCleanup(sched *scheduler.Scheduler, t *testing.T) { schedulerCacheIsEmpty := func() (bool, error) { snapshot := sched.Cache().Snapshot() return len(snapshot.Nodes) == 0 && len(snapshot.AssumedPods) == 0, nil } if err := wait.Poll(time.Second, wait.ForeverTestTimeout, schedulerCacheIsEmpty); err != nil { t.Errorf("Failed to wait for scheduler cache cleanup: %v", err) } } </ins>

While lookint into , I spent few minutes looking into this test: And there is a problem that is relatively easy to reproduce. The problem is that we are running its test cases one by one. But nothing in the meantime is ensuring that the current state that the test assumes is reflected in scheduler. In particular the problem that I observed couple time is that: we finished one test case we call cleanupNodes [but we don't ensure that scheduler actually cleaned up this node] we start the next test case we try to schedule a pod that has some toleration (but shouldn't fit on our node) by accident the toleration make it fit on to the previous node so scheduler schedules it on the node from previous test case
/assign


Question: nodeIPValidator: validateNodeIP, clock:           clock.RealClock{}, enableControllerAttachDetach:            kubeCfg.EnableControllerAttachDetach, <del> iptClient:                               utilipt.New(utilexec.New(), utildbus.New(), utilipt.ProtocolIpv4), </del> <ins> iptClient:                               utilipt.New(utilexec.New(), utildbus.New(), protocol), </ins> makeIPTablesUtilChains:                  kubeCfg.MakeIPTablesUtilChains, iptablesMasqueradeBit:                   int(kubeCfg.IPTablesMasqueradeBit), iptablesDropBit:                         int(kubeCfg.IPTablesDropBit),

/kind bug What happened: Kubelet uses "iptables" instead of "ip6tables" in an ipv6-only cluster. This causes failed traffic for services (and probably a lot of other problems). For instance the chain is created in but is missing in ; When a is assigned the fails; What you expected to happen: All components should use in a ipv6-only cluster. should work in an ipv6-only cluster. How to reproduce it (as minimally and precisely as possible): Start a ipv6-only cluster Check iptables Optional; try a service type: LoadBalancer Anything else we need to know?: is hard-coded in finc in "" Environment: Kubernetes version (use ):Cloud provider or hardware configuration: kvm VMs on Ubuntu 16.04 OS (e.g. from /etc/os-release): Own BusyBox system Kernel (e.g. ):Install tools: None Others:
/sig network
/area ipv6
Thanks for reporting this problem. I'll take a look.
is a link to the code with the hard-coded ipv4 protocol.
Thanks, I will send a pr to fix this issue.
/assign
I see this issue too. Also there is some related discussion with issue .
I just pulled the latest from master, and gave a try with IPv6 cluster, but I still see the error, when using kube-proxy + IPVS: E0829 16:51:56. 1 ] Failed to execute iptables-restore: exit status 2 (ip6tables-restore v1.4.21: The protocol family of set KUBE-LOOP-BACK is IPv4, which is not applicable. Were you able to try this with kube-proxy+IPVS?
Sorry, I didn't try ipv6 cluster with ipvs proxier. Those two issue are related, but not exactly the same issue. Seems we already have conclusion in , nobody working on that?