Question: golang.org/x/exp v0.0.0-20200207192155-f17229e696bd/go.mod h1:J/WKrq2StrnmMY6+EHIKF9dgMWnmCNThgcyBT1FY9mM= golang.org/x/exp v0.0.0-20200224162631-6cc2880d07d6/go.mod h1:3jZMyOhIsHpP37uCMkUooju7aAi5cS1Q23tOzKc+0MU= golang.org/x/exp v0.0.0-20200331195152-e8c3332aa8e5/go.mod h1:4M0jN8W1tt0AVLNr8HDosyJCDCDuyL9N9+3m7wDWgKw= <ins> golang.org/x/exp v0.0.0-20210220032938-85be41e4509f h1:GrkO5AtFUU9U/1f5ctbIBXtBGeSJbWwIYfIsTcFMaX4= golang.org/x/exp v0.0.0-20210220032938-85be41e4509f/go.mod h1:I6l2HNBLBZEcrOoCpyKLdY2lHoRZ8lI4x60KMCQDft4= </ins> golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js= golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0= golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=

For structured logging migration we have developed a tool to klog invocations For a given directory logcheck validates two things: directory is fully migrated - check if there are non-structured logging calls like usage of and is correct - check if key values passed are paired. We run logcheck as part of Kubernetes verify scripts on directories listed in Directories listed in are marked as migrated to avoid regressions from organized migration, however there are also other code locations that already use new functions that are not yet ready to be marked as migrated (components that are not part of organized migrations are done as best effort). We should split checking if directory was migrated from correctness of new functions and scan whole codebase for correctness. Plan: Add a flag to logcheck for requiring that directory was migrated. Add to verify-strucured- running logcheck of whole k/k codebase to check correctness This will ensure higher quality of migration PRs by validating correctness of directories that were not yet in .structuredlogging /wg structured-logging /cc
I'd like to pick this up. Do I need to wait till it is triage/accepted?
/triage accepted
Feel free to pick it up :P Usually I don't want to self triage, but this issue is purely Structured Logging WG related.
/assign
[Case 2: Correct usage] If the directory is not marked as migrated, we check that it's usage of is correct and don't care about usage of . By default, CI will test for correctness of whole k/k codebase instead of testing for complete migration (which is the current behavior). And to test for migration, a new flag must be passed explicitly. is this the expectation?
To clarify, let me rephrase. Current state is What we want: [Case 2: Correct usage] tested on whole repository
If this is the case, I can start implementing it.
SG, thanks for the help.
/milestone v1.22 Part of work go graduate Structured Logging KEP to beta.
I don't think work here has finished. We still need to implement those changes in k/k repo: Cut new klog Bump klog in k/k Update the script ./hack/verify-structured- /reopen
Reopened this issue. <detailsIn response to : I don't think work here has finished.

We still need to implement those changes in k/k repo: Update the script /reopen </details
Why do we need to cut a new klog and bump in k/k? Can't we make latest logcheck available to k/k without it? I don't know the process.
Hmm, We could try without updating to update. Please send PR that will update ./hack/verify-structured- If updating klog is nesesery feel free to contact for help
If we do both, we'd have to run logcheck twice (with and without flag) which means test time will increase.
This depends on implementation, increasing running time is ok as long it's reasonable, for example x1.5, but not x10.
Due to v1.22 code freeze, I'm moving this issue to the next release. /milestone v1.23
We would like to have static analysis setup before we start migrating logs in 1.23. will you be able to work on it this week? Do you need any help?
Yes, I can. PR kubernetes/kubernetes/pull/ is up for enabling static analysis. It has some unresolved conversations. It'd be helpful if I could get another round of review on that PR. Before we merge that we'd need to fix existing logging issues which are listed here: If I could get help fixing those it'd be great. Else I can fix it too.
I see two issues that need to be addressed: Fix handling of in klog code, this should be done by creating an issue to klog () Fix remaining errors Upgrade klog version to pick up fix for Exit


Question: } <ins> func TestCleanLegacyRealServersExcludeCIDRs(t *testing.T) { ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake() ipset := ipsettest.NewFake(testIPSetVersion) gtm := NewGracefulTerminationManager(ipvs) fp := NewFakeProxier(ipt, ipvs, ipset, nil, []string{"4.4.4.4/32"}) fp.gracefuldeleteManager = gtm vs := &utilipvs.VirtualServer{ Address:   net.ParseIP("4.4.4.4"), Protocol:  string(v1.ProtocolUDP), Port:      56, Scheduler: "rr", Flags:     utilipvs.FlagHashed, } fp.ipvs.AddVirtualServer(vs) rss := []*utilipvs.RealServer{ { Address:      net.ParseIP("10.10.10.10"), Port:         56, ActiveConn:   0, InactiveConn: 0, }, { Address:      net.ParseIP("11.11.11.11"), Port:         56, ActiveConn:   0, InactiveConn: 0, }, } for _, rs := range rss { fp.ipvs.AddRealServer(vs, rs) } fp.netlinkHandle.EnsureDummyDevice(DefaultDummyDevice) fp.netlinkHandle.EnsureAddressBind("4.4.4.4", DefaultDummyDevice) fp.cleanLegacyService( map[string]bool{}, map[string]*utilipvs.VirtualServer{"ipvs0": vs}, map[string]bool{"4.4.4.4": true}, ) fp.gracefuldeleteManager.tryDeleteRs() remainingRealServers, _ := fp.ipvs.GetRealServers(vs) if len(remainingRealServers) != 2 { t.Errorf("Expected number of remaining IPVS real servers after cleanup should be %v. Got %v", 2, len(remainingRealServers)) } } </ins> func TestCleanLegacyService6(t *testing.T) { ipt := iptablestest.NewFake() ipvs := ipvstest.NewFake()

What happened: When I config kube-proxy with excludeCIDRs, kube-proxy still try to delete my ipvs rules. What you expected to happen: kube-proxy not delete excludeCIDRs virturl server. How to reproduce it (as minimally and precisely as possible): kubeadm init --config=kubeadm- --experimental-upload-certs Then add a ipvs on node: Then join the node, you can see kube-proxy log: If don't have an active connection, the ipvs will removed. Environment: Kubernetes version (use ): v1.14.0 OS (e.g: ): centos7.4 Kernel (e.g. ): 10 Install tools: kubeadm
/sig network
/triage unresolved I am a bot run by
Kube-proxy delete RS from VIP, even if VIP in excludeCIDRs Because RS are the same with RS from Service VIP's Expected: But kube-proxy make that: - -
I think this is not we expected. I want using ipvs as a local porxy for kubernetes HA.
I've raised the pr
/remove-triage unresolved


Question: } }) <del> It("should create matching services in underlying clusters", func() { </del> <ins> It("should create and update matching services in underlying clusters", func() { </ins> fedframework.SkipUnlessFederated(f.ClientSet) service = createServiceOrFail(f.FederationClientset, nsName, FederatedServiceName) defer func() { // Cleanup

As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.


Question: continue } <del> // Versions and feature support // ============================ // Windows version == Windows Server, Version 1709,, Supports both sandbox and non-sandbox case // Windows version == Windows Server 2016   Support only non-sandbox case // Windows version < Windows Server 2016 is Not Supported // Sandbox support in Windows mandates CNI Plugin. // Presence of CONTAINER_NETWORK flag is considered as non-Sandbox cases here // Todo: Add a kernel version check for more validation if networkMode := os.Getenv("CONTAINER_NETWORK"); networkMode == "" { // On Windows, every container that is created in a Sandbox, needs to invoke CNI plugin again for adding the Network, // with the shared container name as NetNS info, // This is passed down to the platform to replicate some necessary information to the new container // // This place is chosen as a hack for now, since ds.getIP would end up calling CNI's addToNetwork // That is why addToNetwork is required to be idempotent // Instead of relying on this call, an explicit call to addToNetwork should be // done immediately after ContainerCreation, in case of Windows only. TBD Issue # to handle this if r.HostConfig.Isolation == kubeletapis.HypervIsolationValue { // Hyper-V only supports one container per Pod yet and the container will have a different // IP address from sandbox. Return the first non-sandbox container IP as POD IP. // TODO(feiskyer): remove this workaround after Hyper-V supports multiple containers per Pod. if containerIP := ds.getIP(c.ID, r); containerIP != "" { return containerIP } } else { // Do not return any IP, so that we would continue and get the IP of the Sandbox ds.getIP(sandboxID, r) } } else { // ds.getIP will call the CNI plugin to fetch the IP if containerIP := ds.getIP(c.ID, r); containerIP != "" { return containerIP } </del> <ins> if containerIP := ds.getIP(c.ID, r); containerIP != "" { return containerIP </ins> } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: On Windows Server with sandbox support the CNI gets called for all the containers within the pod including the sandbox container itself. What you expected to happen: The CNI should be called only for the sandbox container since they share the same network details. How to reproduce it (as minimally and precisely as possible): kubelet Pod at the parameters that get passed when the CNI is called Anything else we need to know?: This applies to Windows Server with sandbox support. The CNI should be called only for the infra container. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig windows /sig network /kind bug /assign /cc
Reopen this because of
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
has been merged now. /close
Closing this issue. <detailsIn response to : has been merged now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
has this issue been fixed? Is the CNI getting called only for the infra container? From what I see, the CNI still gets called for every single container from within a pod:
See comments It is required for windows containers.
The way I understand it, it's that the DNS issue will be fixed in Windows RS5 and there will not be required additional calls. The workaround at the moment is to call the CNI multiple times until the DNS issue gets fixed in Windows. As far as I know, that is the only reason why the additional calls to the CNI are required.
Is this true? Do we still need to invoke cni for each container in RS5? And even with new cni dns capabilities ()?

the PR that you linked it's unrelated to this issue. The CNI should be called for the sandbox pod which is the infra container. All the other containers created within the same pod are sharing all network details, excepting DNS. This is an issue in 1709 and 1803 where not everything gets replicated properly to the other containers. As a workaround for this, you need to call AttachEndpoint from HCS on the newly created container network interface only to replicate the DNS as well. You can also see more details about this issue here:
Can you re-open the issue in the meanwhile since this issue still exists?
I still think this isn't a issue, but a requirement for windows containers. Even after RS5, we should probably support old windows versions. WDYH
Why would the spec of the CNI should be different on Windows? On Linux you know that the sandbox ID that will get passed to the CNI is always the sandbox container. On Windows the sandbox ID won't be only that, it will also be an ID of a container from within the same pod. Does this sound like normal behavior to you? For now I agree, the workaround is to call the CNI for every single container because of the Windows issue with DNS. But this doesn't mean we have to stick with this for future Windows releases which will not require calling the CNI ADD multiple times. Also, this is a hack for now () which needs to be fixed. Please re-open the issue until we get feedback from others as well, thanks.
OK. Reopened the issue. Let's wait a while for feedbacks from other folks.
Apologize for the delay. Windows doenst have the namespace support native on the platform. we this support as part of RS5 Platform release. We are working on making runtime spec changes and CRI/ContainerD changes to use this model. here is a runtime spec PR that got merged. This is orthogonal to the DNS issue. Once we are done with the CRI/ContainerD changes then if u use CRI/containerD as runtime and if your OS version is RS5 and above then you don't need this second CNI add call. But as Pengfei said, if you are on a downlevel version or using dockershim as Runtime then you will have to make the second CNI call.

So in my opinion this code will stay as workaround forever until we remove the support for downlevel OS. Hope this helps. I would recommend closing this issue.
thanks for the confirmation. In my opinion this should stay open until it gets fixed at least for future Windows releases, because the way it is at the moment, the CNI will still be called multiple times regardless of the Windows version. Why would you recommend closing this issue? Shouldn't this be fixed for futures Windows releases? The other calls will be useless when Windows gets proper namespace support. Also, I recommend watching this presentation about the CNI which had place at the KubeCon from May I don't see why we shouldn't have this open until the issue gets fixed for future releases and we stick with the official SPEC of the CNI.
and I had talked before about end of life of downlevel OS like 1803 and 1709. I think we should keep this open or open a new Issue to track updating the code so that the CNI is not called multiple times
This code would exists as far as we support 1709, 1803. The new workflow to use the windows namespace is getting implemented in containerd/cri. We can move kubernetes to use this new flow only for RS5 and above, if we plan to do it here as well. There isn't a good way to switch between 2 workflows of cni callflow wrt the Windows version hence we are not planning to change here till 1709 & 1803 exists, without adding some ugly code.
thanks for the explanation
I totally agree that this has to be fixed for future release and we are actively working on it. I was recommending to close this issue and open a feature to track this request. We should be able to raise a PR in another 2 weeks to support native network namespace support. This code will stay as long as we support down-level OS'es. I am fine either way. I am trying to close any open issues that is not a bug on Windows Networking(Platform).
/kind feature Although from a kubernetes standpoint, this isn't a new feature. It doesn't need a new KEP and won't change any API. It will just be a future Windows network code cleanup.
/kind cleanup
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
- can you clarify a few things so we know what's required to fix this? 's the minimum OS version needed? Windows Server 2019, or a future release? this be possible to fix for dockershim, or just CRI-containerD? Thanks!
What's the minimum OS version needed? Windows Server 2019, or a future release? [Dinesh]: Minimum OS version needed is 2019 Will this be possible to fix for dockershim, or just CRI-containerD? [Dinesh]: yes its possible to fix for Dockershim as well. but we are starting it with ContainerD. ContainerD is already done and available for testing.
excellent. thank you Dinesh!
ping - you have logs coming in email for this. Can you help us determine if it's needed for 1.14 vs 1.15?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Ping
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
Status on this?


Question: msg, handlerErr := m.runner.Run(kubeContainerID, pod, container, container.Lifecycle.PostStart) if handlerErr != nil { m.recordContainerEvent(pod, container, kubeContainerID.ID, v1.EventTypeWarning, events.FailedPostStartHook, msg) <del> m.killContainer(pod, kubeContainerID, container.Name, "FailedPostStartHook", nil) </del> <ins> if err := m.killContainer(pod, kubeContainerID, container.Name, "FailedPostStartHook", nil); err != nil { glog.Errorf("Failed to kill container %q(id=%q) in pod %q: %v, %v", container.Name, kubeContainerID.String(), format.Pod(pod), ErrPostStartHook, err) } </ins> return msg, ErrPostStartHook } }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I tried to use virtlet with Kubernetes release 1.6.6, but a pod with a VM in it gets stuck in "ContainerCreating" state. This problem does not happen with Kubernetes release 1.6.3. The log of the responsible kubelet shows a panic when trying to restart the container. I do not know why the container is being restarted, but that should not cause a panic in any case. Here is an excerpt from the kubelet's log: It is easy to see a code path that can cause this panic. The offending line in is this: and there is indeed a way for to be nil at this point. Earlier in that function we see the line and there is no intervening statement that checks whether is ; ends with . What you expected to happen: I expected the virtlet to run, like it does with Kubernetes release 1.6.3. How to reproduce it (as minimally and precisely as possible): That would take some work, I hope the diagnosis above makes this moot. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: "bare metal" (really VMs, but who cares?) OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS Kernel (e.g. ): Linux n11rdu1ctrl 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: I installed Kubernetes and virtlet using some local Ansible Others: I am using a local fork of virtlet
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
, did you do any update action against that Pods?

I upped the kubelet's to 4, and got an interesting clue when I tried the experiment again. Here is an excerpt from the more verbose log. Note the empty container name in the "Killing unwanted container" entry. That telling log entry comes from these lines in which is soon followed by .
I only created the pod, no updates.
Is there a running container with id something like when the panic occurs? A quick fix is to add checking whether containerSpec is nil as you mentioned. But it's weird that gets a container without name.
Did virtlet set correctly? Seems it is "", which caused the problem.
Create a PR () that checking containerSpec to avoid panic. And it seems there's no UT case for , I'll create related PRs for
The actual Docker containers are all gone now. But here is what I got from shortly before opening this issue: I will look into the question asked.


Question: func RestartApiserver(cs clientset.Interface) error { // TODO: Make it work for all providers. if !ProviderIs("gce", "gke", "aws") { <del> return fmt.Errorf("unsupported provider: %s", TestContext.Provider) </del> <ins> return fmt.Errorf("unsupported provider for RestartApiserver: %s", TestContext.Provider) </ins> } if ProviderIs("gce", "aws") { initialRestartCount, err := getApiserverRestartCount(cs)

is failing with "framework.RestartControllerManager() = unsupported provider: gke; want nil" /kind failing-test
/sig network
Introduced in , should be fixed by


Question: // The first successful sync with the underlying store. // The cache must wait until this first sync is completed to be deemed ready. // Since we cannot send a bookmark when the lastProcessedResourceVersion is 0, <del> // we poll aggressively for the first RV before entering the dispatch loop. </del> <ins> // we poll aggressively for the first list RV before entering the dispatch loop. </ins> lastProcessedResourceVersion := uint64(0) if err := wait.PollUntilContextCancel(wait.ContextForChannel(c.stopCh), 10*time.Millisecond, true, func(_ context.Context) (bool, error) { <del> if rv := c.watchCache.getResourceVersion(); rv != 0 { </del> <ins> if rv := c.watchCache.getListResourceVersion(); rv != 0 { </ins> lastProcessedResourceVersion = rv return true, nil }

sig-release-master-blocking ci-kubernetes-unit 05/30/2024 19:25 IST Prow Logs: Triage: No response /sig
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
/sig api-machinery
/cc This started last week, but it seems to be fairly rare, so might be a consequence of something a bit earlier. The value of "2" suggests that it's happening on the first event, and my brain goes towards: - will you be able to take a look at it?
sure, I will have a look. /assign
We chatted with about that offline and it seems like a real correctness issue. Regression indeed introduced by We have a race with setting RV in watchcache and incoming events. The current explanation is: initializeds with RV=N event comes with RV=N+1 [but wasn't yet dispatched] polling introduced in that PR catches RV=N+1 then the event (with RV=N+1) is received
So it is a release blocker, then? (Upcoming alpha 1 release cut scheduled for tomorrow)
I don't know what the release policy is but we might have a potential fix for this issue.
This is a real correctness bug introduced 2 weeks ago. I'm not sure what's our policy is for alpha releases, but it sounds like a potential blocker to me.
The fix for it is already approved.


Question: RBDPool:      "pool1", RBDImage:     "image1", FSType:       "ext4", <ins> ReadOnly:     true, </ins> }, }, }),

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I created a job with parallelism=2, 1 container in each pod, and a rbd volume attached to the container. Attachment was read-only. Only 1 pod got the RBD mounted while the other POD failed with "still in use" in syslog, referencing the code in that checks watchers. Note that this is not the same thing as this: which refers to persistent volumes. This is not a persistent volume -the use case is to map read-only data to multiple containers from a single RBD, like for using a reference data set, etc. According to the documentation this should work (see below). What you expected to happen: I expect this to happen per the documentation: "A feature of RBD is that it can be mounted as read-only by multiple consumers simultaneously. This means that you can pre-populate a volume with your dataset and then serve it in parallel from as many Pods as you need. Unfortunately, RBD volumes can only be mounted by a single consumer in read-write mode - no simultaneous writers allowed." How to reproduce it (as minimally and precisely as possible): Create 2 pods, each with 1 container mapping a single rbd read only. Anything else we need to know?: Environment: Kubernetes version (use ): 4 Cloud provider or hardware configuration: self, x8664 systems OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ): Linux dal1util6p30 4.4.0-116-generic -Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: ceph Others: I believe that in the case where a read-only mapping is being requested, the code should not check watchers. Again this is consistent with the documentation but it doesn't work that way. If a PV/PVC is the "right" way to do this it seems seriously overkill and the management overhead is severe, especially in a large scale environment.

The documentation states that the advantage of RBD is to be able to attach read-only volumes to multiple containers concurrently, and this is how I believe it should work. The watcher check is artificial and really needs to apply only to read/write. I can fix this bug myself but would like to understand the reasoning behind it a bit more. Thanks! Leo Reiter
/sig
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reiterating the mentions to trigger a notification: <detailsIn response to : /sig Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I will check the problem.
/assign


Question: # limits the size of metadata fields to 32K, and stripping comments is the # easiest way to buy us a little more room. function prepare-startup-script() { <del> sed '/^s*#([^!].*)*$/ d' ${KUBE_ROOT}/cluster/gce/configure-vm.sh > ${KUBE_TEMP}/configure-vm.sh </del> <ins> # Find a standard sed instance (and ensure that the command works as expected on a Mac). SED=sed if which gsed &>/dev/null; then SED=gsed fi if ! ($SED --version 2>&1 | grep -q GNU); then echo "!!! GNU sed is required.  If on OS X, use 'brew install gnu-sed'." exit 1 fi $SED '/^s*#([^!].*)*$/ d' ${KUBE_ROOT}/cluster/gce/configure-vm.sh > ${KUBE_TEMP}/configure-vm.sh </ins> }

Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): 1.5 beta 2 Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): CVM Kernel (e.g. ): Install tools: kube- What happened: Step 6 of fails What you expected to happen: Node upgrade to succeed. How to reproduce it (as minimally and precisely as possible): Follow the manual upgrade steps outlined in the linked document. Anything else do we need to know: The error is because the metadata is too large for the new node instance template.
More complete error output:
This is blocking test and per email I'm bumpting this to P0.
Does it mean that "node-kube-" is too large? It seems like that too me. If so, we should understand why those are different than on startup...
This is broken for me too: After an upgrade.
: so who is going to be the assignee? and come up with a plan of attack? :)
The problem is that configure- is too large. In my directory: You can see that the number of bytes in configure- () matches the output in the error message (). What is interesting is that configure- in the repo is actually larger_. In the 1.5 branch: and in the 1.4 branch: But we aren't seeing an error during new cluster creation in either release branch.
In the directory when creating a new cluster from the 1.4 branch: which is below the size limit.
Here's the diff between the two configure- scripts: which makes it appear that is the culprit.
This may be a mac issue -- when I manually run the command from the function it makes no changes. But if I run instead of then it strips a bunch of comments, reducing the file size below the allowed limit:
We should be able to do the same thing we do in this file to fix it.
I created but haven't had a chance to test it yet.

(Master and) node upgrade worked for me on Ubuntu, so this does indeed seem to be Mac-specific.
I tested and it looks like it fixed my issue. I suppose I'll need to cherry pick it into the 1.5 branch once it merges to master.


Question: } else { // This never happens, because dedupDeltas never returns an empty list // when given a non-empty list (as it is here). <del> // But if somehow it ever does return an empty list, then // We need to remove this from our map (extra items in the queue are // ignored if they are not in the map). delete(f.items, id) </del> <ins> // If somehow it happens anyway, deal with it but complain. if oldDeltas == nil { klog.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; ignoring", id, oldDeltas, obj) return nil } klog.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; breaking invariant by storing empty Deltas", id, oldDeltas, obj) f.items[id] = newDeltas return fmt.Errorf("Impossible dedupDeltas for id=%q: oldDeltas=%#+v, obj=%#+v; broke DeltaFIFO invariant by storing empty Deltas", id, oldDeltas, obj) </ins> } return nil }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing another PR, I noticed incorrect commentary and code. The primary offense is at , where a comment asserts (incorrectly) that a key can be present in but not and the code removes a key from without ensuring it is absent from . The only saving grace is that this code is actually unreachable (as also noted in the comment). I think it would be better to fix the comment and in this unreachable situation, than to violate an invariant that is critical to other code in this file. also contemplates the possibility that a key can be in and absent from . What you expected to happen: I expected there to be no comments or code that contemplate violations of the invariant at How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): current branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Sorry, I should have noticed this in my earlier pass over this code.
Agreed. Was discussing this with at
We discussed this one, Mike will send PR, Daniel will review.


Question: <ins> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail set -o xtrace export REPO_DIR=${REPO_DIR:-$(pwd)} # Produce a JUnit-style XML test report for Jenkins. export KUBE_JUNIT_REPORT_DIR=${WORKSPACE}/_artifacts # Run the kubekins container, mapping in docker (so we can launch containers), # the repo directory, and the artifacts output directory. # # Note: We pass in the absolute path to the repo on the host as an env var incase # any tests that get run need to launch containers that also map volumes. # This is required because if you do # # $ docker run -v $PATH:/container/path ... # # From _inside_ a container that has the host's docker mapped in, the $PATH # provided must be resolvable on the *HOST*, not the container. docker run --rm=true  -v /var/run/docker.sock:/var/run/docker.sock  -v "$(which docker)":/bin/docker  -v "${REPO_DIR}":/go/src/k8s.io/kubernetes  -v "${KUBE_JUNIT_REPORT_DIR}":/workspace/artifacts  --env REPO_DIR="${REPO_DIR}"  -it gcr.io/google_containers/kubekins-test:0.1  bash -c "cd kubernetes && ./hack/jenkins/test-dockerized.sh" </ins>

In the pull request Jenkins, we're running up to 8 instances of verification checks, build, unit tests, integration tests, and e2e tests. Unfortunately, it seems like several of these components were not designed to run simultaneous to other tests on the same machine. In particular, several unit tests seem to write to static file names in /tmp (or equivalent), which is a fairly easy fix. Some of the verification checks and the integration test start up etcd, however, and the etcd helper checks that only a single etcd binary is running on the host (and uses a static port for it). For example: We might be able to fix the etcd issue separately, or we can just figure out how to containerize all of these tests so they can't possibly interfere with each other. Note that we're ignoring the exit status of the verification checks and unit/integration tests right now, so these etcd failures are effectively being ignored (though it is causing us to miss test coverage).
What percentage of tests are failing due to being non-hermetic like this?
I'm not sure about unit tests (I've anecdotally seen just a few fail), but the verification checks are failing fairly regularly, and the integration test is also similarly frequently not running due to this.
To be clear, I'm looking for real numbers, to decide whether it's feasible to temporarily disable a small number of tests until we can hermetify them. Otherwise we need to turn off all unit, integration and verification tests in Jenkins to avoid irritating invalid regression signals. We need to do one or the other this week.
I'll try to dig through the logs, figure out which ones have failed recently, disable them in Jenkins, and see whether the flakes abate.
I just turned off all unit, integration and verification checks in Jenkins, as a temporary measure to unstall the merge-bot.
As a point of information, I just found approximately 20 tests in which use a static tmp path with semi-random filenames which are highly likely to collide. We could keep searching for unit tests that are non-hermetic, but it'd probably be a better investment of time to fix the test infrastructure so test authors don't need to care (and so we don't need to go look at and fix ~2000 test cases).

Another few metrics, using the string "/tmp" as a likely anti-pattern: We'd need to update each of these 23 files to get them to use or if they aren't already.
Created to track cleaning up tempdir usage in unit tests.
cc


Question: // ... HPAScaleToZero: {Default: false, PreRelease: featuregate.Alpha}, <del> DisableKubeletCSRAdmissionValidation: {Default: false, PreRelease: featuregate.Deprecated}, // remove in 1.33 </del> <ins> AllowInsecureKubeletCertificateSigningRequests: {Default: false, PreRelease: featuregate.Deprecated}, // remove in 1.33 </ins> StorageNamespaceIndex: {Default: true, PreRelease: featuregate.Beta},

This comes from We've a feature gate, . However, we try to name feature gates after the positive thing they add, even if it's on by default, or even if we're setting if false because the positive thing is being dropped (rare, but it happens). Typically we don't start feature gate names with , as it makes the docs jar: Example quote 1: To temporarily revert to the legacy behavior, enable the feature gate. vs Example quote 2: You can enable the feature gate if you run code that relies on the previous behavior. You should plan to migrate. and because disabling things isn't really a feature. Some examples of gates you can or once could disable by turning them off_ include: - - - So, I recommend a rename of the gate before the v1.31 release. /sig auth architecture
FYI
Example gate names: (off by default) (off by default) (on by default) (on by default) (on by default) from (off by default, deprecated)
/milestone v1.31
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.31 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Another possibility: , deprecated, off by default
Hello there! I am new to kubernetes and I want to contribute in it. Is it something that I can contribute to?
This issue is not a good issue for a first-time contributor to work on.
I'm a fan of , deprecated, off by default
hoping one of you are doing the PR? :) /milestone v1.31


Question: { description: "notready sandboxes without containers for deleted pods should be garbage collected.", sandboxes: []sandboxTemplate{ <del> makeGCSandbox(pods[2], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, false), </del> <ins> makeGCSandbox(pods[2], 0, runtimeapi.PodSandboxState_SANDBOX_NOTREADY, false, 0), </ins> }, containers: []containerTemplate{}, remain:     []int{},

kubernetes pre-submit unit tests for pass some times and fails some times causing the flakiness. Here is one such failure https://k8s- Passing: Failing: cc
Yep, I'm checking with this
Sorry for introducing a flakiness. Sent out the fix in .
/assign
Dup of . /close

https://k8s- Failed: TestSandboxGC {kuberuntime}
https://k8s- Failed: TestSandboxGC {kuberuntime}
/assign
Should've been fixed by


Question: return ss, nil } <ins> func (ss *scaleSet) getVMSS(vmssName string, crt cacheReadType) (*compute.VirtualMachineScaleSet, error) { getter := func(vmssName string) (*compute.VirtualMachineScaleSet, error) { cached, err := ss.vmssCache.Get(vmssKey, crt) if err != nil { return nil, err } vmsses := cached.(*sync.Map) if vmss, ok := vmsses.Load(vmssName); ok { result := vmss.(*vmssEntry) return result.vmss, nil } return nil, nil } vmss, err := getter(vmssName) if err != nil { return nil, err } if vmss != nil { return vmss, nil } klog.V(3).Infof("Couldn't find VMSS with name %s, refreshing the cache", vmssName) ss.vmssCache.Delete(vmssKey) vmss, err = getter(vmssName) if err != nil { return nil, err } if vmss == nil { return nil, cloudprovider.InstanceNotFound } return vmss, nil } </ins> // getVmssVM gets virtualMachineScaleSetVM by nodeName from cache. // It returns cloudprovider.InstanceNotFound if node does not belong to any scale sets. func (ss *scaleSet) getVmssVM(nodeName string, crt cacheReadType) (string, string, *compute.VirtualMachineScaleSetVM, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Recently we upgraded our cluster from 1.14.4 to 1.14.7. Then we started to observe tons of following errors: I guess the cause may be related to this line which was introduced recently: As we have hundreds of services with LoadBalancer type, and we also enable cluster-autoscaler, the nodes list change very often. For each service, there will be one call to GetScaleSet. The quota will be exhausted quickly. When fails, new nodes created by cluster-autoscaler will lose connectivity to load balancer. What you expected to happen: Kubernetes should not hit VMSS throttling threshold. How to reproduce it (as minimally and precisely as possible): a cluster with aks-engine. Use VMSS. 200 services with LoadBalancer type. up VMSS to 50 nodes down VMSS to 60 nodes scaling up and down VMSS Anything else we need to know?: Environment: Kubernetes version (use ): 1.14.7 Cloud provider or hardware configuration: Azure, VMSS OS (e.g: ): Ubuntu Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /area provider/azure
I create a PR to mitigate the issue.
do we have a rough num about how this PR would mitigate this issue? looks like we need to cherry-pick this PR to old releases. it's better if we have a rough estimation.
The revised version of code calls instead of azure API. The cache refreshes itself every 10 min. I believe this will solve the throttling issue.
I believe we're being affected by this. Apart from getting the - when creating our to use a pre-configured Azure Public IP, the ingress appears to be getting a different IP each time
Thank you all!
The PR is merged. Does your issue remain?

pls cherry-pick that fix to old release, thanks
Cherry picked to release 1.14-1.16
I’ll retry today and let you know Wysłane z iPhone'a
Issue returned in K8S 1.15.4 Warning UpdateLoadBalancerFailed service/nginx-ingress-controller Error updating load balancer with new hosts map[<nodes names]: [EnsureHostInPool(ingress/nginx-ingress-controller): backendPoolID(/subscriptions/<subscription/resourceGroups/<rg/providers/Microsoft.Network/loadBalancers/<rg/backendAddressPools/<rg) - failed to ensure host in pool: "azure - cloud provider rate limited(read) for operation:VMSSGet", EnsureHostInPool(ingress/nginx-ingress-controller):


Question: ) const ( <del> maxKubectlExecRetries = 5 </del> <ins> maxKubectlExecRetries           = 5 DefaultNamespaceDeletionTimeout = 5 * time.Minute </ins> ) // Framework supports common operations used by e2e tests; it will keep a client & a namespace for you.

/assign The test flakes presubmit testing in PRs six times per week The site can focus on just this test: Which shows that 83 out of 85 errors match this pattern:
Previous attempt to address this flake:
CC
Took a quick look at a failed test: https://k8s- The indicates: . or and can you help investigate.
CC
The failure in the GKE multizone test suite is probably due to the GKE issue I fixed last week.
We'll dig into this and see what's up.
Expectations: This work should interrupt and take priority over other activity while it is assigned to you. Expectation for sig leads is to identify someone to fix the issue, them and yourself. Expectation for assignees are to investigate the issue and either a) resolve it or b) it to a more appropriate person and b) yourself.
I repeatedly ran this test on my e2e cluster since 7pm last night, and it has not failed yet. Seems like it fails more frequently in the PR env instead of the CI env, so may need to look into differences between the two. The test is failing during the deletion of the namespace. It seems like there are still some resources not getting deleted. I'm trying to dig deeper in the logs to see if I can find what resources are not getting cleaned up.
Here's one failed : From the API server initial namespace deletion request from the e2e test: I0427 23:23:29. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (4.744775ms) 200 [[ (linux/amd64) kubernetes/] 104.197.101.155:] controller triggers deletion on all the objects in the namespace: I0427 23:24:37. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-cx3bh/statefulsets: (91.176059ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] I0427 23:24:37.

5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh/resourcequotas: (7.608267ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] etc ... gets finally deleted by controller I0427 23:26:30. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-cx3bh/statefulsets: (87.184389ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serv iceaccount:kube-system:namespace-controller] [::1]:] I0427 23:26:30. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh/resourcequotas: (21.517822ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccoun t:kube-system:namespace-controller] [::1]:] etc ... deletion finished I0427 23:26:31. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (16.58598ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] 2E test cleanup tries to delete it again: I0427 23:26:31. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (9.104808ms) 404 [[ (linux/amd64) kubernetes/] 104.197.101.155:] <= This is the e2e test failure cleanup Total time to delete 3 mins Here's a successful : : deletion request I0428 00:45:14. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (11.208829ms) 200 [[ (linux/amd64) kubernetes/] 104.197.153.246:] starts object deletions I0428 00:45:46.

5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-c4t8j/deployments: (33.104152ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:servi ceaccount:kube-system:namespace-controller] [::1]:] I0428 00:45:46. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j/podtemplates: (3.447937ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:k ube-system:namespace-controller] [::1]:] etc ... gets finally deleted by controller I0428 00:46:17. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j/podtemplates: (10.679029ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount: kube-system:namespace-controller] [::1]:] I0428 00:46:20. 5 ] DELETE /apis/extensions/v1beta1/namespaces/e2e-tests-pv-c4t8j/deployments: (6.130617ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system: serviceaccount:kube-system:namespace-controller] [::1]:] etc ... deletion is finished I0428 00:46:21. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (21.10359ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:na mespace-controller] [::1]:] 2E test cleanup I0428 00:46:32.

5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (1.436939ms) 404 [[ (linux/amd64) kubernetes/] 104.197.153.246:] Total time to delete ~1 minute Comparing the bad and good runs, these were some supicious areas: the bad run, the period between 1) and 2) was 1 minute, whereas in the good run it was 30 seconds the bad run, the period between 2) and 3) was ~2 minutes, whereas in the good run it was ~20 seconds. The slowness in the bad run seems systematic, I did not find any one object that was taking a particularly long time to delete, all of them seemed to be taking longer compared to the good run. I'm not sure where to dig down further to find the cause of the general slowness. I don't know if this is expected, and we should just increase the timeout? The test framework seems to have a default timeout of 5 minutes, whereas this volume test timeout is only 3 minutes.
Great analysis! Since no singular event stands out as taking too much time how about, in the name of expediency, just changing the timeout to 5m and see how the tests behave?
The delay of ns deleteion might related to this PR and . is still open.
Those two issues do not seem to impact the deletion of the namespace itself, so I don't think fixing those will solve this issue.
But I think delaying the deletion of a pod will affect the time to delete the namespace.


Question: var prober *etcd3ProberMonitor clientErr := fmt.Errorf("etcd client connection not yet established") <del> go wait.PollUntil(time.Second, func() (bool, error) { newProber, err := newETCD3ProberMonitor(c) </del> <ins> go wait.PollImmediateUntil(time.Second, func() (bool, error) { </ins> lock.Lock() defer lock.Unlock() <ins> newProber, err := newETCD3ProberMonitor(c) </ins> // Ensure that server is already not shutting down. select { case <-stopCh:

flakes several times recently /kind flake /sig api-machinery
This failure occurs for all tests in as mentioned in The description shows flaking.
/cc
/cc I remember someone tried to fix it some time ago,
/assign for context of /triage accepted
All of them fail with "connection not yet established" There is something odd in these tests, since we mock the KV but we create a "real" etcd process, I wonder if we flake on trying to connect to the real etcd process or there is some weird race because of that
do you want to try with ?


Question: return fmt.Errorf("failed to watch %s, err: %v", path, err) } case mode&os.ModeSocket != 0: <del> w.wg.Add(1) go func() { defer w.wg.Done() w.fsWatcher.Events <- fsnotify.Event{ Name: path, Op:   fsnotify.Create, } }() </del> <ins> event := fsnotify.Event{ Name: path, Op:   fsnotify.Create, } //TODO: Handle errors by taking corrective measures if err := w.handleCreateEvent(event); err != nil { klog.Errorf("error %v when handling create event: %s", err, event) } </ins> default: klog.V(5).Infof("Ignoring file %s with mode %v", path, mode) }

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: "apiVersion": "v1", "metadata": map[string]interface{}{ "generateName": nil, <ins> "generation":   nil, </ins> "labels": map[string]interface{}{ "foo": nil, },

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRDs created with old Kubernetes version are missing : Example: CRD created recently: $ curl -s '…' {creationTimestamp, generation}' { "creationTimestamp": "2018-07-23T17:24:58Z", "generation": 1 } CRD created with one of the old versions: $ curl -s 'https://…' {creationTimestamp, generation}' { "creationTimestamp": "2018-04-11T12:38:47Z", "generation": null } What you expected to happen: I'd expect to be always present so I can use it in controllers and monitoring without resorting to ugly hacks. How to reproduce it (as minimally and precisely as possible): Create a CRD with an old version of Kubernetes with CRD subresources disabled (not sure which one, 1.10 or maybe 1.9). Update the cluster to 1.11. Create another CRD. Observe that is always present in the new one, but doesn't show up in the old one. Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T18:02:47Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): CoreOS 1800.7.0 Kernel (e.g. ): 4.14.63-coreos Install tools:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Custom resources start setting as 1 during creation from v1.10 because that's when the subresources feature was introduced. The subresources feature says that I want to follow the spec/status conventions and hence, want generation to be initialized to 1 and be incremented whenever spec changes. In <=v1.9, even if we initialized it to 1, it does not hold much meaning because is not allowed to be modified during updates. It is also not incremented at all, since we have no way of knowing if the user wants to follow spec/status conventions. So it will always remain as 1 and won't be helpful in controllers. The condition is surprising. In the older version, if it is absent with , it should also be absent with . I am not able to reproduce this though. would you know on which version you tested the export against?
/area custom-resources /cc
I don't expect anything from old clusters, that's completely understandable. However, if I have a 1.11 cluster with subresources enabled, I expect all CRDs to behave in the same way, not just the ones that were created after 1.10. This doesn't seem to be the case. Version information is in the original issue. All our clusters are currently running v1.11.3. I think that some of the buggy CRDs were created with v1.10 (without CRD subresources enabled), but it might've been v1.9.
I think the older cluster version here is 1.9 because we always set generation to 1 during creation from 1.10: 10 did change the default to 1. However, I am not sure if we should be changing the default values for a 1.9 cluster now. Moreover, we maintain only 3 releases at a time - so I think we only cherry-pick to 1.12, 1.11 and 1.10 now (?). thoughts?
What I'm trying to say is that clusters running 1.11 should automatically handle resources with missing or unset by updating it to 1. It doesn't make sense to keep them in this broken state forever just because they were created in one of the previous versions.

Actually, after editing the resource the is updated to 1, so I can just default to in the controller. For some reason I thought that the resource stays broken forever, which is not the case. Sorry about that!
We can still do this, but I'm wondering if that's ok or not because that wouldn't represent the "true" generation value for that resource. Have the same questions for , which initializes and increments generation for all other resources. :thinking:
I've updated the issue since export is not relevant (and I think the logic resets generation to 1). However, the fact that for old CRDs is (internally) set to 0 is still problematic. Any custom controllers or monitoring code that checks will not be able to tell whether they've been processed successfully by a controller or not (because both and are missing).
/assign
Are you suggesting we set generation on read if it is missing?
That would be one option, or just patch them once after update.
it would probably make sense to make this part of metadata coercion on read, based on whether the CRDs has spec/status enabled
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Hello! I'm bug triage lead for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16?
No idea, I'm not a maintainer.
Yes


Question: out.MinSyncPeriod = in.MinSyncPeriod out.Scheduler = in.Scheduler out.ExcludeCIDRs = *(*[]string)(unsafe.Pointer(&in.ExcludeCIDRs)) <ins> out.StrictARP = in.StrictARP </ins> return nil }

/kind bug Kubernetes version: 1.13.1. kube-proxy mode: IPVS. SDN: flannel. We are used services like this: where IP 192.168.10.201 is from our internal /24 network (in particular, our Kube nodes have IPs 192.168.10.1, 192.168.10.2 etc). Note that there is no conflicts between real IPs and virtual external ones. Before updating from 1.12.3 to 1.13.1, it worked correctly. In partucular these IPs was accessible from other non-Kubernetes hosts from 192.168.10.0/24 network. After update kube-proxy to 1.13.1, this feature stops working: non-Kubernetes hosts cannot resolve service IPs via ARP: On Kubernetes hosts it still working, because not requires ARP resolving. If we reset sysctl parameter arpignore on Kube hosts, all works normally even from non-Kube hosts. Quck debugging shows that problem is in the commit . Please make this behaviour opt-out. (Now we resets arpignore to 0 via cron every 1 min.)
/sig network
Having the same problem on Kubernetes 1.13.2 btw Thanks for a workaround!
Same problem, fix this please!!!
This is similar to We've had a discussion there on how to best implement a solution but haven't decided yet I'm in favor of flag which defaults to the previous behavior to avoid breaking existing setups (alternatives are cleaner but involve creating an additional arptable rule per service which I think we should avoid) what do you think?
OK

What happened: After PR: , we noticed communications issue in our pods. We use the Lyft CNI plugin () which relies on two interfaces in pod: an IPVLAN interface connecting to the AWS VPC which has a VPC IP an Unnumbered point-to-point interface (veth) connecting to the host and that does not have an IP (it "borrows" the one from the IPVLAN interface) On the host side, there is a veth interface which also doesn't have an IP (it borrows one from the main interface). With the latest versions of kube-proxy communication between host and pods break What you expected to happen: Communication still works How to reproduce it (as minimally and precisely as possible): Use the Lyft CNI plugin Deploy kube-proxy Try to ping the host from a pod (or a pod from the host) Anything else we need to know?: The new sysctls create 2 issues: containers started after kube-proxy inherit so they will not answer ARP queries on the veth interface the veth on the host side also does not have an IP so it will no answer ARP queries from the pods I know that our setup is pretty uncommon but assuming that the veth on the host side will have an IP seems dangerous. Some discussions on the PTP plugin for instance suggest not using IP for the host interface: containernetworking/plugins Possible fixes: create a flag to enable ARP filtering (I think to avoid breaking existing setups, we should disable it by default but I'm fine with enabling it by default as long as we document the solution) use arptables to filter ARP queries for clusterIPs (): a bit more work because we need to do it for all clusterIPs (there may be a more efficient way to do it but I could not make it work by filtering on the kube-ipvs interface) and this requires the arptables binary. We could test for the existence of the binary and log a warning if it is missing (similar to behavior with conntrack) Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign ping
I can say only that solution with arptables working fine with MetalLB.
Oh, really? Glad you confirm that.

But I would like to make sure that we won't create too many arptables rules. Can ipset works with arptables?
It doesn't look like arptables works with ipset
Unfortunately doesn't support ipsets, but new do. (not fully implemented yet) Eg we can create ipset for arp: But current implementation is not allowing to use it for arp rules for now: Will return: I guess because there is still no type specified for (destination ip) and (source ip) BTW: nft does not allow to create set of integer components: But there is a way for specify multiple IPs in one rule, eg: Where is IPv4 addresses in decimal format More info here:
Another option would be to use the and filter all queries but this flag is not in kube-proxy today
Bad idea, it will handle but not - which can be really differ
can you explain more detail why Lyft CNI can not assign IPs to the right interfaces for avoid this kind of issues?
Yes you are right. bad idea It could definitely add IPs to these interfaces but it's not the choice that was made. More details here: ("Unnumbered point-to-point interface"). I could probably PR a fix there. I'm a bit worried that some other CNI plugins could break here though
We are about to deploy some 1.13 clusters and can't use kube-proxy 1.13 as is We could start with a custom build but I assume others will be impacted The simplest way forward for now is probably to use a flag to control this behavior. I think the safest is probably to disable this by default to avoid the issue for people upgrading but if you think it is really important we can enable it by default. I know we'd like to avoid adding more flags but other solutions seem a bit more complicated and will most likely result in one iptables rule per service. What do you think ?
Just FYI, we have another CNI plugin that is affected by this.
We addressed the issue by assigning a link-local address 169.254.1.1/32 to the host-side veth in . So this issue is not critical at least for our case.
Good news everyone, new nft supports setting ip address for arp filters : that's makes possible for using sets


Question: // is different than parallelism, otherwise the previous controller loop // failed updating status so even if we pick up failure it is not a new one exceedsBackoffLimit := jobHaveNewFailure && (active != *job.Spec.Parallelism) && <del> (int32(previousRetry)+1 > *job.Spec.BackoffLimit) </del> <ins> (failed > *job.Spec.BackoffLimit) </ins> if exceedsBackoffLimit || pastBackoffLimitOnFailure(&job, pods) { // check if the number of pod restart exceeds backoff (for restart OnFailure only)

What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): $oc create job -f $ oc get job jobname -ojson | jq .status { "active": 1, "failed": 1, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 2, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 3, "startTime": "2020-07-30T06:45:37Z" } { "active": 1, "failed": 4, "startTime": "2020-07-30T06:45:37Z" } { "conditions": [ { "lastProbeTime": "2020-07-30T06:53:10Z", "lastTransitionTime": "2020-07-30T06:53:10Z", "message": "Job has reached the specified backoff limit", "reason": "BackoffLimitExceeded", "status": "True", "type": "Failed" } ], "failed": 5, "startTime": "2020-07-30T06:45:37Z" } $ oc get job jobname -ojson | jq .spec.backoffLimit 6 Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: 4.5.3 Server Version: 4.4.13 Kubernetes Version: v1.17.1+ Cloud provider or hardware configuration: OS (e.g: ): NAME="Red Hat Enterprise Linux CoreOS" VERSIONID="4.4" OPENSHIFTVERSION="4.4" RHELVERSION="8.2" ID="rhcos" IDLIKE="rhel fedora" ANSICOLOR="0;31" CPENAME="cpe:/o:redhat:enterpriselinux:8::coreos" HOMEURL="" BUGREPORTURL="" REDHATBUGZILLAPRODUCT="OpenShift Container Platform" REDHATBUGZILLAPRODUCTVERSION="4.4" REDHATSUPPORTPRODUCT="OpenShift Container Platform" REDHATSUPPORTPRODUCTVERSION="4.4" OSTREEVERSION='44.82.-0' Kernel (e.g.

): Linux ip-10-0-192-232 4.18.0-193.13.2.el82.x8664 SMP Mon Jul 13 23:17:28 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Others: apiVersion: batch/v1 kind: Job metadata: name: sre-build-test-fail namespace: openshift-build-test spec: backoffLimit: 6 completions: 1 parallelism: 1 template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: matchExpressions: key: node- operator: In values: infra containers: command: /bin/bash -c | # ensure we fail if something exits non-zero set -o errexit set -o nounset set -o pipefail exit 1 imagePullPolicy: Always name: sre-build-test restartPolicy: Never terminationGracePeriodSeconds: 30 tolerations: effect: NoSchedule key: node- operator: Equal value: infra
/sig apps
not sure but that is the best I can think of at this time
i have noticed this problem also, and sometimes, it seems to work normally.
so I found out that in the test is so this can cause the issue.. it has been there from the beginning of the codes inception.. WDYT?
/assign
Duplicate of . /close
Closing this issue. <detailsIn response to : Duplicate of . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Name:      daemonSetName, }, Spec: apps.DaemonSetSpec{ <ins> Selector: &metav1.LabelSelector{ MatchLabels: labelSet, }, </ins> Template: v1.PodTemplateSpec{ ObjectMeta: metav1.ObjectMeta{ Labels: labelSet,

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: In : Which test(s) are failing: - Since when has it been failing: At least 2019-01-07 Testgrid link: Reason for failure: /sig apps /sig cluster-lifecycle
cc
/assign


Question: } apt-get-install() { <ins> local -r packages=( $@ ) installed=true for package in "${packages[@]}"; do if ! dpkg -s "${package}" &>/dev/null; then installed=false break fi done if [[ "${installed}" == "true" ]]; then echo "== ${packages[@]} already installed, skipped apt-get install ${packages[@]} ==" return fi apt-get-update </ins> # Forcibly install packages (options borrowed from Salt logs). until apt-get -q -y -o DPkg::Options::=--force-confold -o DPkg::Options::=--force-confdef install $@; do echo "== install of packages $@ failed, retrying =="

I noticed that there is a huge regression on node bootup since Wednesday last couple of days: Bad one: vs. good one: Please note that the issue is not related to any kubernetes component. The node hasn't run salt client yet. This explains a lot of node not ready issue we have lately. cc/
cc/
Whoa, it shouldn't even be running at all. That implies someone a package recently.
I didn't see any new package is , and the entire package size is the same. :-(
Can you dump /var/log/apt.history (I think it is) into here?
Sorry, . Hmm. I don't see anything new on a recent-ish head node, either, but YMMV.
Yes, just about to do it ... Here you are, and please note that it is from a different node with the same bad symptom:
Yeah, no issue there. Oh. Hm. Sorry, I forgot the one directly in .
Wait. The one in configure-vm is new, it's from , and we need to take it out.
Oh no - so sorry :-( does add an apt-get update to configure-vm. But you are installing bridge-utils from apt, so I thought it was correct to apt-get update first. I can easily remove the apt-get-update and add it back into the AWS version instead. In my experience, crazy-slow apt-get problems are either due to an IPv6 address being served to an IPv4 only host, or getting a bad mirror. Is there a GCE internal mirror? Could also be the gluster repo as well - not necessarily the debian repo.
Probably the bestest thing would be to make sure bridge-utils is on the base image, only apt-get-install bridge-utils if brctl is not available, and run apt-get-update as part of apt-get-install only once per script invocation. But this isn't actually fixing the problem. Something is seriously wrong if an apt-get update takes 12 minutes.
is actually built-in on (the later is a no-op, sorry). I went through an effort at some point to make sure that never needed to or anything, because we were having troubles like this.
I can work up a PR that will probably work for all parties.
Yes, that is the root cause and we want to remove all external dependencies at bootup time. Assign this to
Thanks & sorry again to all.


Question: } <ins> // Test cases: // index    finalizer    timestamp    wantLB  |  clean-up //   0         0           0            0     |   false    (No finalizer, no clean up) //   1         0           0            1     |   false    (Ignored as same with case 0) //   2         0           1            0     |   false    (Ignored as same with case 0) //   3         0           1            1     |   false    (Ignored as same with case 0) //   4         1           0            0     |   true //   5         1           0            1     |   false //   6         1           1            0     |   true    (Service is deleted, needs clean up) //   7         1           1            1     |   true    (Ignored as same with case 6) </ins> func TestNeedsCleanup(t *testing.T) { testCases := []struct { desc               string

What happened: Service Load balancer may not be cleaned up on type change if below sequence happen: type changes from to controller restarts/crashes before it gets a chance to process the update from 1. controller starts up again and sees the service doesn't want load balancer, hence skips it. load balancer resource sticks around even if it is no longer needed by the service. Note that this is a different issue than , as load balancer resource will still be cleaned up when the service object is deleted, just not when the type change happens. Anything else we need to know?: This is a corner case that is not covered by the initial service load balancer finalizer implementation (). This can probably be fixed in a trivial way --- make sure the handler takes in service with non-LoadBalancer type but has finalizer attached: I will put more thoughts later. cc
/sig network
Can i reserve this issue in advance? :) I'd like to take this issue as my first issue.
Sure thing, feel free to work on this and glad that you can help :)
Do you mean attach finalizer for service that non-LoadBalancer type in AddFunc? I'm new to kubernetes, hope you can guide me. :)
No worries. I actually meant should include service with non-LoadBalancer type && with finalizer attached. As we only attach finalizer to a service only when we are creating a load balancer for it, a finalizer presents on the service (no matter what type it has) indicates there is a load balancer associated. Hence it makes sense that we should clean up the load balancer if the service no longer wants one, even if the service has not been deleted.
To clarify further, "attach finalizer for service that non-LoadBalancer type in AddFunc" would not be what we want --- we shouldn't attach finalizer to service that we will not be creating load balancer for.
After go through service controller part code, i guess you mean add a condition for as follows: . Am i right? I'm not sure when service type changes if can be involved. Can you clarify this ?
if ok && (wantsLoadBalancer(svc) (svc.Spec.Type != v1.ServiceTypeLoadBalancer && servicehelper.HasLBFinalizer(svc))) . Yea this was what I meant.

On service type change, will be involved instead of . That part is already handled. This issue is explicit for the case when controller fails to handle the type change update before it exiting (e.g. crashing, restart, etc.). Putting such detection to is more off a remedy, it means the next time service controller starts up, it can re-syncs all existing services and find out it still need to cleanup load balancer for this service, even if it doesn't have load balancer type anymore.
Thanks . I have sent a PR assined to you. I think it should be checked in function . It is the case that need to clean up whatever the reason is.


Question: } firstRunning.Wait() <ins> sendSignals() </ins> fakeFilter.wait() firstRunning.Add(concurrentRequests) for i := 0; i < concurrentRequests; i++ { go func() { defer wg.Done()

Testgrid doesn't load kubernetes/test-infra From: /sig api-machinery
/assign /triage accepted
Yes. It looks like a bug. There are many calls to , and they are not all guaranteed to happen before the second . I will revise.


Question: importpath = "k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1", visibility = ["//visibility:public"], deps = [ <ins> "//pkg/scheduler/api:go_default_library", </ins> "//pkg/scheduler/apis/config:go_default_library", "//pkg/scheduler/internal/cache:go_default_library", "//pkg/scheduler/util:go_default_library",

What would you like to be : a default normalize score function in the scheduling framework. the plugin and log an error when the normalizing phase returns an invalid range of scores. Why is this needed: Per discussion in , we should enforce the range of score returned by the scoring plugin. /sig scheduling /priority important-soon /assign


Question: // to set values and determine whether its allowed AdmissionControl      admission.Interface CorsAllowedOriginList []string <ins> HSTSDirectives        []string </ins> // FlowControl, if not nil, gives priority and fairness to request handling FlowControl utilflowcontrol.Interface

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add the "Strict-Transport-Security" header to the API server responses. Why is this needed: Nessus scanner treats this as a Medium Risk in kubernetes 1.18.10.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
/remove-sig api-machinery /sig auth


Question: checkPodStatus(t, kl, podToAdmit, v1.PodPending) } <ins> func TestPodResourceAllocationReset(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.InPlacePodVerticalScaling, true)() testKubelet := newTestKubelet(t, false) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet kubelet.statusManager = status.NewFakeManager() nodes := []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{Name: testKubeletHostname}, Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("8"), v1.ResourceMemory: resource.MustParse("8Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("4"), v1.ResourceMemory: resource.MustParse("4Gi"), v1.ResourcePods:   *resource.NewQuantity(40, resource.DecimalSI), }, }, }, } kubelet.nodeLister = testNodeLister{nodes: nodes} cpu500m := resource.MustParse("500m") cpu800m := resource.MustParse("800m") mem500M := resource.MustParse("500Mi") mem800M := resource.MustParse("800Mi") cpu500mMem500MPodSpec := &v1.PodSpec{ Containers: []v1.Container{ { Name: "c1", Resources: v1.ResourceRequirements{ Requests: v1.ResourceList{v1.ResourceCPU: cpu500m, v1.ResourceMemory: mem500M}, }, }, }, } cpu800mMem800MPodSpec := cpu500mMem500MPodSpec.DeepCopy() cpu800mMem800MPodSpec.Containers[0].Resources.Requests = v1.ResourceList{v1.ResourceCPU: cpu800m, v1.ResourceMemory: mem800M} cpu800mPodSpec := cpu500mMem500MPodSpec.DeepCopy() cpu800mPodSpec.Containers[0].Resources.Requests = v1.ResourceList{v1.ResourceCPU: cpu800m} mem800MPodSpec := cpu500mMem500MPodSpec.DeepCopy() mem800MPodSpec.Containers[0].Resources.Requests = v1.ResourceList{v1.ResourceMemory: mem800M} cpu500mPodSpec := cpu500mMem500MPodSpec.DeepCopy() cpu500mPodSpec.Containers[0].Resources.Requests = v1.ResourceList{v1.ResourceCPU: cpu500m} mem500MPodSpec := cpu500mMem500MPodSpec.DeepCopy() mem500MPodSpec.Containers[0].Resources.Requests = v1.ResourceList{v1.ResourceMemory: mem500M} emptyPodSpec := cpu500mMem500MPodSpec.DeepCopy() emptyPodSpec.Containers[0].Resources.Requests = v1.ResourceList{} tests := []struct { name                          string pod                           *v1.Pod existingPodAllocation         *v1.Pod expectedPodResourceAllocation state.PodResourceAllocation }{ { name: "Having both memory and cpu, resource allocation not exists", pod:  podWithUIDNameNsSpec("1", "pod1", "foo", *cpu500mMem500MPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "1": map[string]v1.ResourceList{ cpu500mMem500MPodSpec.Containers[0].Name: cpu500mMem500MPodSpec.Containers[0].Resources.Requests, }, }, }, { name:                  "Having both memory and cpu, resource allocation exists", pod:                   podWithUIDNameNsSpec("2", "pod2", "foo", *cpu500mMem500MPodSpec), existingPodAllocation: podWithUIDNameNsSpec("2", "pod2", "foo", *cpu500mMem500MPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "2": map[string]v1.ResourceList{ cpu500mMem500MPodSpec.Containers[0].Name: cpu500mMem500MPodSpec.Containers[0].Resources.Requests, }, }, }, { name:                  "Having both memory and cpu, resource allocation exists (with different value)", pod:                   podWithUIDNameNsSpec("3", "pod3", "foo", *cpu500mMem500MPodSpec), existingPodAllocation: podWithUIDNameNsSpec("3", "pod3", "foo", *cpu800mMem800MPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "3": map[string]v1.ResourceList{ cpu800mMem800MPodSpec.Containers[0].Name: cpu800mMem800MPodSpec.Containers[0].Resources.Requests, }, }, }, { name: "Only has cpu, resource allocation not exists", pod:  podWithUIDNameNsSpec("4", "pod5", "foo", *cpu500mPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "4": map[string]v1.ResourceList{ cpu500mPodSpec.Containers[0].Name: cpu500mPodSpec.Containers[0].Resources.Requests, }, }, }, { name:                  "Only has cpu, resource allocation exists", pod:                   podWithUIDNameNsSpec("5", "pod5", "foo", *cpu500mPodSpec), existingPodAllocation: podWithUIDNameNsSpec("5", "pod5", "foo", *cpu500mPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "5": map[string]v1.ResourceList{ cpu500mPodSpec.Containers[0].Name: cpu500mPodSpec.Containers[0].Resources.Requests, }, }, }, { name:                  "Only has cpu, resource allocation exists (with different value)", pod:                   podWithUIDNameNsSpec("6", "pod6", "foo", *cpu500mPodSpec), existingPodAllocation: podWithUIDNameNsSpec("6", "pod6", "foo", *cpu800mPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "6": map[string]v1.ResourceList{ cpu800mPodSpec.Containers[0].Name: cpu800mPodSpec.Containers[0].Resources.Requests, }, }, }, { name: "Only has memory, resource allocation not exists", pod:  podWithUIDNameNsSpec("7", "pod7", "foo", *mem500MPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "7": map[string]v1.ResourceList{ mem500MPodSpec.Containers[0].Name: mem500MPodSpec.Containers[0].Resources.Requests, }, }, }, { name:                  "Only has memory, resource allocation exists", pod:                   podWithUIDNameNsSpec("8", "pod8", "foo", *mem500MPodSpec), existingPodAllocation: podWithUIDNameNsSpec("8", "pod8", "foo", *mem500MPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "8": map[string]v1.ResourceList{ mem500MPodSpec.Containers[0].Name: mem500MPodSpec.Containers[0].Resources.Requests, }, }, }, { name:                  "Only has memory, resource allocation exists (with different value)", pod:                   podWithUIDNameNsSpec("9", "pod9", "foo", *mem500MPodSpec), existingPodAllocation: podWithUIDNameNsSpec("9", "pod9", "foo", *mem800MPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "9": map[string]v1.ResourceList{ mem800MPodSpec.Containers[0].Name: mem800MPodSpec.Containers[0].Resources.Requests, }, }, }, { name: "No CPU and memory, resource allocation not exists", pod:  podWithUIDNameNsSpec("10", "pod10", "foo", *emptyPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "10": map[string]v1.ResourceList{ emptyPodSpec.Containers[0].Name: emptyPodSpec.Containers[0].Resources.Requests, }, }, }, { name:                  "No CPU and memory, resource allocation exists", pod:                   podWithUIDNameNsSpec("11", "pod11", "foo", *emptyPodSpec), existingPodAllocation: podWithUIDNameNsSpec("11", "pod11", "foo", *emptyPodSpec), expectedPodResourceAllocation: state.PodResourceAllocation{ "11": map[string]v1.ResourceList{ emptyPodSpec.Containers[0].Name: emptyPodSpec.Containers[0].Resources.Requests, }, }, }, } for _, tc := range tests { if tc.existingPodAllocation != nil { // when kubelet restarts, AllocatedResources has already existed before adding pod err := kubelet.statusManager.SetPodAllocation(tc.existingPodAllocation) if err != nil { t.Fatalf("failed to set pod allocation: %v", err) } } kubelet.HandlePodAdditions([]*v1.Pod{tc.pod}) allocatedResources, found := kubelet.statusManager.GetContainerResourceAllocation(string(tc.pod.UID), tc.pod.Spec.Containers[0].Name) if !found { t.Fatalf("resource allocation should exist: (pod: %#v, container: %s)", tc.pod, tc.pod.Spec.Containers[0].Name) } assert.Equal(t, tc.expectedPodResourceAllocation[string(tc.pod.UID)][tc.pod.Spec.Containers[0].Name], allocatedResources, tc.name) } } </ins> func TestHandlePodResourcesResize(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.InPlacePodVerticalScaling, true)() testKubelet := newTestKubelet(t, false)

After enabling In-Place Pod Vertical Scaling, if deploy a pod without setting memory (or cpu) request, kubelet will be failed at its second restart kubelet restart successfully kubelet with In-Place Pod Vertical Scaling enabled a pod without setting memory request, e.g: kubelet for the first time kubelet for the second time will fail with error: After deploying the pod, the following information is saved in the file : Then restart kubelet for the first time, the relevant content in the file becomes: When deploying the pod, as there are no records in the checkpoint (), kubelet will get resource config from directly and save it to the checkpoint (without memory request) When restarting kubelet for the first time, it will restore the previously saved data from the checkpoint. Since the memory request was not previously set, it will be set to an empty value and resaved to the checkpoint When restarting kubelet for the second time, it will process as a value of 0 (not empty), which resulting in the inconsistency between the calculated checksum during data recovery and the previous one <details</details<details</details<details</details<details</details<details</details<details</details
/sig node
cc
/triage accepted
/assign
xlink: /cc
thanks for finding this issue and the fix. It looks good with suggestion. For beta, I wonder if we can do away with local checkpoint and rely on PodStatus as persistent store and source of truth for allocated resources. It will simplify the implementation.
Using resource.Quantity in checkoutPoint is a bad usage. I found except the case in this issue, whenever adding a static pod to the kubelet, then restart the kubelet, restore the resize status checkpoint will fail, casing kubelet panic. The reason is , before restart kubelet, the checkpoint struct in kubelet will looks like: and the the podstatusmanager_state file is looks like: and after restart kubelet and recover the checkoutpoint state, the checkpoint struct in kubelet will looks like: the difference is between "kube-scheduler:map[cpu:{i:{value:6 scale:-1} d:{Dec:<nil} s: Format:DecimalSI}" and "kube-scheduler:map[cpu:{i:{value:600 scale:-3} d:{Dec:<nil} s:600m Format:DecimalSI} ".

It's caused by json Marshal and Unmarshal of resource.Quantity. So, I think Using resource.Quantity in checkoutPoint is a bad case. We should consider using PodStatus in apiserver instead of checkoutPoint.
Is this problem solved? Now I can still reproduce this problem in the official version of 1.28. Thanks~
Could anyone please let me know what is the state of the art to solve this issue?
we have seen the similar problem after enabling InPlaceUpdate pod resources and could we reopen this ?
can we just do marshal&unmarshal before storing state checkpoint to avoid the issue of resource.Quantity?
I am +1 on doing away with the checkpointing code altogether and leveraging PodStatus. We discussed this in the Kubernetes Contributor Summit in Chicago. I just have not been able to find the cycles to work on it :(


Question: t.Fatalf("bad cidr: %v", err) } cfg.ExtraConfig.ServiceIPRange = *cidr <ins> cfg.GenericConfig.PublicAddress = netutils.ParseIPSloppy("2001:db8::10") </ins> _, secCidr, err := netutils.ParseCIDRSloppy(secondaryServiceCIDR) if err != nil {

Clusters created with kubeadm using DualStack are not able to work, because CoreDNS pods never get ready Seeing one failed job logs It seems that the pod is "waiting for the kubernetes service"?? coredns version seems the same as in the working jobs CoreDNS pod should be ready Instal a Dual-stack cluster using master with kubeadm (or kind ) ! It started to fail on Oct12 <details</details<details</details<details</details<details</details<details</details<details</details
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc I didn't notice it was failing for more than 1 month :facewithhead_bandage: I didn't have time to dig into it yet, but this PR landed just that day how is that the pods running has the version 1.8.4 but the PR seems to update kubeadm to 1.8.6?
Looking into this today. The error from CoreDNS should only happen when the kubernetes plugin fails to sync with the Kubernetes API, either because it can’t connect, or it can’t sync the lists of services/endpointslices. There were changes and fixes to the API connection code, so it could be a regression.
Actually no. The changes I was thinking of were already in 1.8.4.
How can we reproduce it : Instal a Dual-stack cluster using master with kubeadm (or kind ) I was unable to reproduce this issue in a kind dual stack cluster.

After upgrading coredns to 1.8.6 (from 1.8.0 installed by kind), and adding list/watch permission for endpointslice, the pods were ready. edit: It also works for coredns 1.8.4 - I'm unclear as to which version I should be checking.
let me check if is a regression in kind :thinking:
working with 1.8.6 and yesterdays kubernetes podIPs
This is weird, I hit a similar issue in openshift, endpoints doesn't match the IP family of the kubernetes.default service apiserver service is single stack, at least until this KEP fixes it What I don't understand is why the order of the IPs in kind is different let me try to fix both things /assign
my fault, sorry for the noise :sweat_smile:


Question: "k8s.io/klog/v2" schedulerserverconfig "k8s.io/kubernetes/cmd/kube-scheduler/app/config" "k8s.io/kubernetes/cmd/kube-scheduler/app/options" <del> "k8s.io/kubernetes/pkg/api/legacyscheme" </del> "k8s.io/kubernetes/pkg/scheduler" kubeschedulerconfig "k8s.io/kubernetes/pkg/scheduler/apis/config" "k8s.io/kubernetes/pkg/scheduler/framework/runtime"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kube-Scheduler exposes metrics in a secure manner (TLS and authentication required) on port by default. If one does not authenticate itself, the error message displayed is: What you expected to happen: Kube-Scheduler should emit a clear and understandable error message if authentication fails. Kube-Controller-Manager does that the following way: How to reproduce it (as minimally and precisely as possible): up a Kubernetes Cluster to access the exposed metrics endpoint (https://metrics-server:/metrics) without authentication Anything else we need to know?: xref: Environment: Kubernetes version (use ): 1.18.6 Cloud provider or hardware configuration: Bare metal OS (e.g: ): Debian GNU/Linux 10 (buster) Kernel (e.g. ): 4.19.0-9-amd64 Install tools: Kubeadm (Kube-Scheduler )
/sig scheduling instrumentation
If ok, I can try this fix /assign
/assign