Question: fs.StringSliceVar(&o.OIDC.SigningAlgs, "oidc-signing-algs", []string{"RS256"}, ""+ "Comma-separated list of allowed JOSE asymmetric signing algorithms. JWTs with a "+ <del> "'alg' header value not in this list will be rejected. "+ </del> <ins> "supported 'alg' header values are: RS256, RS384, RS512, ES256, ES384, ES512, PS256, PS384, PS512. "+ </ins> "Values are defined by RFC 7518 https://tools.ietf.org/html/rfc7518#section-3.1.") fs.Var(cliflag.NewMapStringStringNoSplit(&o.OIDC.RequiredClaims), "oidc-required-claim", ""+

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --A kube-apiserver does not support all of the signing algorithms listed in RFC 7518 It fails with something like When looking into the or in the code I really get the impression that all algorithms from the RFC document are supported But after some time, I noticed the most essential word is : The documentation to be improved to explicitly list the supported algorithms instead of referring to the RFC document that lists much more algorithms than the actual supported ones. Run the kube-apiserver with flag I think the really supported algorithms are explicitly listed here Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig auth /kind documentation
: I will be happy to do it, but would require some more insight on what is to be done?
Hi Could you please elaborate what is not clear? I think I have provided quite detailed information and references to the code and documentation. Just to be clear, the documentation is correct even today, but it is hard to digest as the referenced RFC document does not describe which signing algorithm is asymmetric and which symmetric.
Hi IMO We can do two things here either we will mention the supported signing algorithm for oidc instead of or explicitly mention the supported signing algorithm in the docs. What do you suggest? I'm up for it if nobody is doing that.
Fixed it here:


Question: } } <ins> // waitForUsedResourceQuota polls a ResourceQuota status for an expected used value func waitForUsedResourceQuota(t *testing.T, c clientset.Interface, ns, quotaName string, used v1.ResourceList) { err := wait.Poll(1*time.Second, resourceQuotaTimeout, func() (bool, error) { resourceQuota, err := c.CoreV1().ResourceQuotas(ns).Get(context.TODO(), quotaName, metav1.GetOptions{}) if err != nil { return false, err } // used may not yet be calculated if resourceQuota.Status.Used == nil { return false, nil } // verify that the quota shows the expected used resource values for k, v := range used { actualValue, found := resourceQuota.Status.Used[k] if !found { t.Logf("resource %s was not found in ResourceQuota status", k) return false, nil } if !actualValue.Equal(v) { t.Logf("resource %s, expected %s, actual %s", k, v.String(), actualValue.String()) return false, nil } } return true, nil }) if err != nil { t.Errorf("error waiting or ResourceQuota status: %v", err) } } </ins> func scale(t *testing.T, namespace string, clientset *clientset.Clientset) { target := int32(100) rc := &v1.ReplicationController{

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --integration test jobs TestQuotaLimitService which was recently in One example run: I suspect this is because TestQuotaLimitService runs through multiple test cases where each test case relies on state created from the previous case. However, resource quota changes take some time to get propagated on the system and the test is trying to validate acceptance of new Services before the resource quota controller applies the necessary changes. links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --
cc
/sig network /sig apimachinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig network /sig apimachinery Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
Opened a fix here
/triage accepted


Question: fw.fakeKubelet, stats.NewResourceAnalyzer(fw.fakeKubelet, time.Minute, &record.FakeRecorder{}), fw.fakeAuth, <del> oteltrace.NewNoopTracerProvider(), </del> kubeCfg, ) fw.serverUnderTest = &server

Follow-up to Discussed in As part of kubelet tracing beta, we should add tracing support to the kubelet's read-only endpoint, which currently does not have tracing. It should use WithPublicEndpoint to prevent callers from influencing sampling decisions. Part of
/sig node /sig instrumentation
/triage accepted /assign
can i help you with this issue?
I don't think we'll be able to start with this until merges
Ok, I'm waiting to see if I can help you. Thank you
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale is merged now, so this can be worked on.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Could I help with this issue?
Sure. See We pass a NoOp Tracer provider to disable tracing for the endpoint. Instead, we need to pass the tracerprovider, but make sure that the eventual uses withPublicEndpoint. Looking closer, i'm not sure its currently possible to use WithPublicEndpoint with the otelrestful package... doesn't contain that option (unlike otelhttp: ), so we may need to add that upstream first.

Another thing to keep in mind is that updating otel libraries is currently blocked by (see my attempt in )
I gave the whole thing another try. At least locally, the dependencies seem now to be able to be updated. :raccoon: Could you maybe have a look at it :)
He's on leave for the time being.
oh, can you provide some feedback or point me to the right people?
I'm back now. Feel free to assign me to PRs related to this, and I'll review
Welcome back :) Could you re-run the CI ? Its just the dependencies. The PR itself shouldn't be that much afterwards.
otel is now up-to-date enough. here:
Hi, I would like to ask about one trouble that caused by WithPublicEndpoint. Aggregated APIServer that I customized uses kubernetes's RecommendedConfig. Due to the fact that WithPublicEndpoint is enabled, it can't be tracked by the full trace for its span-link model. And I think child-parent span model is more appropriate. So, could you please give me some advice? Looking forward to your reply.
thanks. We know about this, and are tracking it in


Question: return "", errors.New(ErrDeviceNotFound) } <ins> realPath, err := filepath.EvalSymlinks(path) if err != nil { return "", fmt.Errorf("failed to evaluate path %s: %s", path, err) } </ins> // losetup -j {path} returns device in the format: // /dev/loop1: [0073]:148662 ({path}) // /dev/loop2: [0073]:148662 (/dev/sdX)

How to reproduce it (as minimally and precisely as possible): /var/lib/kubelet a symlink kubelet a block PVC + a pod that uses it. the pod. -The pod is forever. In kubelet logs: The reason is that shows: kubelet can't find there, because losetup shows real path . Environment: Kubernetes version (use ): 1.18 /sig storage /priority important-soon
/triage accepted


Question: local host_platform host_platform=$(kube::golang::host_platform) <del> # Use eval to preserve embedded quoted strings. </del> local goflags goldflags goasmflags gogcflags <del> eval "goflags=(${GOFLAGS:-})" </del> goldflags="${GOLDFLAGS:-} -s -w $(kube::version::ldflags)" goasmflags="-trimpath=${KUBE_ROOT}" gogcflags="${GOGCFLAGS:-} -trimpath=${KUBE_ROOT}"

What happened: When building kubernetes from source for 1.10 it was possible to limit the parallel linker calls using GOFLAGS="-p 1". (I understand that the new format is GOFLAGS="-p=1" so I am using that) With kubernetes 1.11 the build aborts: What you expected to happen: The build should complete, using just one process for linking. How to reproduce it (as minimally and precisely as possible): export GOFLAGS="-p=1" make -j1 Anything else we need to know?: The environment is quite memory constrained, therefore preventing the build to fork a linker for each CPU is required. Any other option instead of the GOFLAGS would work as well. Environment: Kubernetes version (use ): 12.0 OS (e.g. from /etc/os-release): Arch Linux ARM Others: go version go1.11.1 linux/arm64
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
This is because a change in go 1.11, see , and related go issues: , etc. I've opened a PR to address this: .
/assign


Question: // undesired becomes completely unused, all the config objects are // read and processed as a whole. <ins> // The funcs in this package follow the naming convention that the suffix // "Locked" means the relevant mutex must be locked at the start of each // call and will be locked upon return.  For a configController, the // suffix "ReadLocked" stipulates a read lock while just "Locked" // stipulates a full lock.  Absence of either suffix means that either // (a) the lock must NOT be held at call time and will not be held // upon return or (b) locking is irrelevant. </ins> // StartFunction begins the process of handling a request.  If the // request gets queued then this function uses the given hashValue as // the source of entropy as it shuffle-shards the request into a

Two flakes recently: They fail with error like: or /assign
/sig scalability
In logs I have found the following slow calls: It's a bug that we use exempt priority level for clusterloader: I created It's quite worrisome that APF handler takes 1s for exempt request.
I checked a APF code and I think the flow for exempt requests is: genericfilters.WithPriorityAndFairness() workEstimate := workEstimator.EstimateWork(r) fcIfc.Handle() cfgCtlr.startRequest() () looks for matching flowschemas in cfgCtrl.flowSchemas returns for exempt for other requests it calculates sha256 for flowDistinguisher and queues request req.Finish updates metrics calls execFn updates headers handler.ServeHTTP which stops the timer for priorityandfairness For me, the most likely reason for slowness is lock contention on We know that the test is super cpu-intensive in that part and other requests are e.g. calculating sha256 keeping that lock. A potential improvement would be to change this lock to RWLock or return the lock before starting calculating sha256 (if we believe that this is the most expensive operation there). /cc
/assign
I think the lock-contention is even worse than what you wrote, as e.g.: is happening under this lock (which is also dispatching requests). I will take a deeper look later this week, but I think that switching to RWLock might do the job without any invasive changes.
may trigger the execution of some requests that were waiting, but they are not executed in the same goroutine. Yes, I think changing that Mutex to a RWMutex would be an easy way to greatly reduce contention.
I agree switching to RWMutex is likely to help a lot, if that's not enough I think with a little work we could e.g. switch to s.
(perhaps because that's what my code originally did? Reviewers thought the retirement dance was too tricky.)
/triage accepted /priority important-soon I will work on the fix.
/sig api-machinery
Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped. /reopen
Reopened this issue. <detailsIn response to : Wow! Thanks a lot for rapid help here! Let's wait for the next test run to see if it helped.

/reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, can you please report the filter latency regardless of whether it is bad or good? I would like to know how much of a difference that PR made.
Sure, will report the latency change in both cases.
I compared (the one where we saw issue) (first with the lock improvements) I tried to find requests that: use exempt PL (so shouldn't be throttled) has fl_priorityandfairness, key which means that priority and fairness filter took more than 100ms Sample matching request: In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! So overall it looks great. The only worrisome metric I see is that actual P&F throttling for other requests has grown: With locking improvements: ! Old: ! E.g. 'node-high' has now throttling latency of ~1.5s in spikes where in the previous test run it was more like ~800ms. I don't think the lock improvements touched this directly. I expect that this is either: per-test variance (if we rerun the old run again we would see different throttling latencies as well) we throttle exempt requests less (as expected) giving less resources to other requests. Anyway, I don't think this change is meaningful as per-test latency aggregates for PATCH nodes/status and PUT leases (the most common requests in node-high) look ~same as previously: ! ! I think we can close this for now as the issue seems to be resolved. Thanks a lot for your help here! /close
Closing this issue. <detailsIn response to : I compared In the old run, I see ~6k such requests, with the locking improvements there are no such requests. The latency of PATCH jobs seems to improve significantly: ! ! ! I expect that this is either: ! ! /close </details
: thanks for reporting!


Question: pod = f.PodClient().Create(pod) framework.ExpectNoError(framework.WaitForPodSuccessInNamespace( f.ClientSet, pod.Name, f.Namespace.Name)) <ins> var err error pod, err = f.PodClient().Get(pod.Name, metav1.GetOptions{}) framework.ExpectNoError(err) </ins> } else { pod = f.PodClient().CreateSync(pod) framework.ExpectNoError(f.WaitForPodReady(pod.Name)) } <ins> // Verify Pod affinity colocated the Pods. loader := getRunningLoaderPod(f) Expect(pod.Spec.NodeName).To(Equal(loader.Spec.NodeName)) </ins> return pod } <del> func createAppArmorProfileCM(f *framework.Framework) (*api.ConfigMap, error) { </del> <ins> func createAppArmorProfileCM(f *framework.Framework) { </ins> profileName := appArmorProfilePrefix + f.Namespace.Name profile := fmt.Sprintf(`#include <tunables/global> profile %s flags=(attach_disconnected) {

The "AppArmor load AppArmor profiles should enforce an AppArmor profile" e2e test in failing on our 5k-node gce clusters - https://k8s- (run no. 8) due to timeout: Seems like the test is trying to load the profiles across all the nodes using a daemonset and then running a test pod to check that. It's passing on our 2k-node clusters (https://k8s-) but failing for 5k - suggesting that it's likely due to size of the cluster. I think it's the ds creation taking longer on large cluster. cc
Seems like you're the most relevant here. Could one of you take a look into it? And does increasing the timeout for the test-pod based on no. of nodes sound reasonable?
Assigning it you since is OOO. Feel free to reassign as apt.
This is part of the correctness suite and large cluster tests are release-blocking.
FYI
Reassigning to who is back on Tuesday. He wrote the AppArmor stuff and should be able to debug this quickly. /assign
Ref
Could you manage to take a look into this?
Sorry, I meant to reply to this! I have a fix, just need to confirm it works. I think you're right about the DaemonSet timing out. There's no reason this needs to load the profiles on every node, so the fix just runs a single loader, and uses pod affinity to co-schedule the test pod.
Smooth!
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package configfiles import ( "fmt" "path/filepath" "strings" "testing" "github.com/davecgh/go-spew/spew" apiequality "k8s.io/apimachinery/pkg/api/equality" "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig" kubeletscheme "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/scheme" kubeletconfigv1alpha1 "k8s.io/kubernetes/pkg/kubelet/apis/kubeletconfig/v1alpha1" utilfiles "k8s.io/kubernetes/pkg/kubelet/kubeletconfig/util/files" utilfs "k8s.io/kubernetes/pkg/kubelet/kubeletconfig/util/filesystem" utiltest "k8s.io/kubernetes/pkg/kubelet/kubeletconfig/util/test" ) func addFile(fs utilfs.Filesystem, path string, file string) error { if err := utilfiles.EnsureDir(fs, filepath.Dir(path)); err != nil { return err } if err := utilfiles.ReplaceFile(fs, path, []byte(file)); err != nil { return err } return nil } func TestLoad(t *testing.T) { kubeletScheme, _, err := kubeletscheme.NewSchemeAndCodecs() if err != nil { t.Fatalf("unexpected error: %v", err) } // get the built-in default configuration external := &kubeletconfigv1alpha1.KubeletConfiguration{} kubeletScheme.Default(external) defaultConfig := &kubeletconfig.KubeletConfiguration{} err = kubeletScheme.Convert(external, defaultConfig, nil) if err != nil { t.Fatalf("unexpected error: %v", err) } cases := []struct { desc   string file   string expect *kubeletconfig.KubeletConfiguration err    string }{ {"empty data", ``, nil, "was empty"}, // invalid format {"invalid yaml", `*`, nil, "failed to decode"}, {"invalid json", `{*`, nil, "failed to decode"}, // invalid object {"missing kind", `{"apiVersion":"kubeletconfig/v1alpha1"}`, nil, "failed to decode"}, {"missing version", `{"kind":"KubeletConfiguration"}`, nil, "failed to decode"}, {"unregistered kind", `{"kind":"BogusKind","apiVersion":"kubeletconfig/v1alpha1"}`, nil, "failed to decode"}, {"unregistered version", `{"kind":"KubeletConfiguration","apiVersion":"bogusversion"}`, nil, "failed to decode"}, // empty object with correct kind and version should result in the defaults for that kind and version {"default from yaml", `kind: KubeletConfiguration apiVersion: kubeletconfig/v1alpha1`, defaultConfig, ""}, {"default from json", `{"kind":"KubeletConfiguration","apiVersion":"kubeletconfig/v1alpha1"}`, defaultConfig, ""}, } fs := utilfs.NewFakeFs() for i := range cases { dir := fmt.Sprintf("/%d", i) if err := addFile(fs, filepath.Join(dir, kubeletFile), cases[i].file); err != nil { t.Fatalf("unexpected error: %v", err) } loader, err := NewFsLoader(fs, dir) if err != nil { t.Fatalf("unexpected error: %v", err) } kc, err := loader.Load() if utiltest.SkipRest(t, cases[i].desc, err, cases[i].err) { continue } // we expect the parsed configuration to match what we described in the ConfigMap if !apiequality.Semantic.DeepEqual(cases[i].expect, kc) { t.Errorf("case %q, expect config %s but got %s", cases[i].desc, spew.Sdump(cases[i].expect), spew.Sdump(kc)) } } // finally test for a missing file desc := "missing kubelet file" contains := "failed to read" loader, err := NewFsLoader(fs, "/fake") if err != nil { t.Fatalf("unexpected error: %v", err) } _, err = loader.Load() if err == nil { t.Errorf("case %q, expect error to contain %q but got nil error", desc, contains) } else if !strings.Contains(err.Error(), contains) { t.Errorf("case %q, expect error to contain %q but got %q", desc, contains, err.Error()) } } </ins>

From in The most important things I want to see through node e2e tests: 1) the kubelet is started from last-known-good-config file 2) kubelet can download the new config properly to the node; and restart itself from the new config file 3) kubelet can mark the config bad properly after several round of crashing caused the bad config 4) Kubelet can roll back the last-known-good-config and restart itself from that good one 5) kubelet can properly reports its configuration status Note: (2) is covered by existing e2enode tests (1, 4, 5) were covered by the , all of which I plan to turn into automated tests after this PR merges - but I'm trying not to increase the size of this PR any further ~(3) we need to figure out how to test this. I don't want to rely on a "valid" config that causes a crash loop - because the very existence of such a config should be viewed as a bug. I also don't want to include a "please crash loop for our tests" field in the config, because that's an ugly API. The best solution is not clear here.~ We decided to remove crash-loop detection, see . Filing this issue to track this work: into e2e_node tests [x] remote config [x] local (from file) config [x] ~test the crash-loop detection~ removed, see above /cc
[MILESTONENOTIFIER] Milestone Removed Important: This issue was missing labels required for the v1.8 milestone for more than 7 days: kind: Must specify exactly one of [, , ]. priority: Must specify exactly one of [, , ]. Removing it from the milestone. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
The test cases in the OP have been covered by our e2e_node tests for a while.


Question: "encoding/hex" "fmt" "net" <ins> "regexp" </ins> "strings" "k8s.io/apimachinery/pkg/api/resource"

What happened: I tried to set up bandwidth limiting by booting a cluster with , which by default sets up kubenet. I used the annotations xM xM and then used to test the ingress and egress bandwidths, with no effect. Does anyone know if this feature is still supported in kubenet? And if so what are the steps I need to take to boot a cluster that can use pod annotations to do tc shaping. What you expected to happen: Per traffic shaping should be implemented on Kubenet as of 4/12/2018. Anything else we need to know?: There are a lot of issues regarding traffic shaping not working. None have a concrete answer of how to get it to work on kubenet: Relevant: I believe this is where it was first implemented. Environment: Kubernetes version (use ): 1.11.8 Cloud provider or hardware configuration: AWS CC:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
/triage unresolved I am a bot run by
can we put this on sig-network agenda? the discussion item would be: what is the current status of this feature (alpha, beta, GA?) what should we do next with this API?
I have this to the agenda, will grab some info.
AFAIK traffic shaping is still supported in kubenet. Does a pod with these annotations come up successfully but has no traffic shaping applied? Or do things fail earlier? If it comes up successfully, could you run ?
On GKE we do not officially support this, our default image even don't have tc installed. I tried with ubuntu, the kubenet is failed to parse the tc output. kubenet expect. // filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 tc return.// filter parent 1: protocol ip pref 1 u32 chain 0 fh 800::800 order 2048 key ht 800 bkt 0 flowid 1:1 notinhw
If this issue has been triaged, please comment .

Meta: /athenabot mark-triage-reminder I am a bot run by
Depending on which CNI plugin you use you need to install the bandwidth plugin, see:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
since it is in progress /priority critical-urgent /milestone v1.17
/lifecycle frozen
/assign
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
The fix is still pending in review.
Will it be merged by this Thursday for v1.17 code freeze?
/milestone v1.18 (v1.17 code freeze)
matching assigned priority on PR
Hello I'm one of the Bug Triage shadows for the 1.18 release. Since this issue is tagged for 1.18, I'd like to check its status. Is this issue intended for milestone 1.18? Friendly reminder: Code freeze is scheduled for March 5th.
Hello Checking in again. Do you think this will make it in for 1.18 release?
Hey Checking in again as this issue is marked as . Are you close to completing this for v1.18 release? Another reminder that code freeze is just around the corner! (March 5th).
Friendly reminder that code freeze is in 2 days and this issue is marked as
The fix got approved by ,should be able to catch the freeze.


Question: // When the queue is empty, invocation of Pop() is blocked until new item is enqueued. // When Close() is called, the f.closed is set and the condition is broadcasted. // Which causes this loop to continue and return from the Pop(). <del> if f.IsClosed() { </del> <ins> if f.closed { </ins> return nil, ErrFIFOClosed }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed.

Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile:


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package v1beta2 import ( "fmt" apps "k8s.io/api/apps/v1beta2" "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels" ) // ReplicaSetListerExpansion allows custom methods to be added to // ReplicaSetLister. type ReplicaSetListerExpansion interface { GetPodReplicaSets(pod *v1.Pod) ([]*apps.ReplicaSet, error) } // ReplicaSetNamespaceListerExpansion allows custom methods to be added to // ReplicaSetNamespaceLister. type ReplicaSetNamespaceListerExpansion interface{} // GetPodReplicaSets returns a list of ReplicaSets that potentially match a pod. // Only the one specified in the Pod's ControllerRef will actually manage it. // Returns an error only if no matching ReplicaSets are found. func (s *replicaSetLister) GetPodReplicaSets(pod *v1.Pod) ([]*apps.ReplicaSet, error) { if len(pod.Labels) == 0 { return nil, fmt.Errorf("no ReplicaSets found for pod %v because it has no labels", pod.Name) } list, err := s.ReplicaSets(pod.Namespace).List(labels.Everything()) if err != nil { return nil, err } var rss []*apps.ReplicaSet for _, rs := range list { if rs.Namespace != pod.Namespace { continue } selector, err := metav1.LabelSelectorAsSelector(rs.Spec.Selector) if err != nil { return nil, fmt.Errorf("invalid selector: %v", err) } // If a ReplicaSet with a nil or empty selector creeps in, it should match nothing, not everything. if selector.Empty() || !selector.Matches(labels.Set(pod.Labels)) { continue } rss = append(rss, rs) } if len(rss) == 0 { return nil, fmt.Errorf("could not find ReplicaSet for pod %s in namespace %s with labels: %v", pod.Name, pod.Namespace, pod.Labels) } return rss, nil } </ins>

As a prerequisite of moving the , , , and controllers to use the apps/v1beta2 APIs, we need to add list expansions for , , , and to .
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig apps


Question: func podWithID(id, desiredHost string) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ <del> Name:     id, UID:      types.UID(id), SelfLink: fmt.Sprintf("/api/v1/%s/%s", string(v1.ResourcePods), id), </del> <ins> Name: id, UID:  types.UID(id), </ins> }, Spec: v1.PodSpec{ NodeName:      desiredHost,

What happened: pod can't be removed from scheduler cache when be deleted, logs in kube-scheduler is below: I1128 16:41:50. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:50. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:53. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:53. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:57. ] successfully acquired lease kube-system/kube-scheduler I1128 16:41:57. ] caches populated E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it What you expected to happen: pod in scheduler cache can be deleted successfully when pod is deleted How to reproduce it (as minimally and precisely as possible): main steps to reproduce this bug is these: scheduler run in leader-elect=true modeschedulerA is the leader scheduler, and schedulerB is an assistant.

pod1 with "namespace1/name1" and it runs on Node1 down the network between schedulerB and kube-apiserver pod1 and create pod2 with same name, ie "namespace1/name1" and pod2 been assigned to Node1 by schedulerA again. off schedulerA and recover the network between schedulerB and kube-apiserver, and then schedulerB become the leader scheduler. pod2. this time log in schedulerB will print this: E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it this time, if I create a new pod3 with podAntiAffinity to pod1. the result is pod3 can't been assigned to Node1 because pod1 is still in schedulerB's scheduler cache. Anything else we need to know?: the reason of this bug is below: since PR(), podKey in scheduler cache use pod's UID instead of namespace/name however, the key in SharedIndexInformer still use namespace/name in this upper case, delete the pod and create a same name pod may be delivered as one update notification.( see disscusion ) scheduler received pod update notification and then print below log. E1128 16:42:11. ]

scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig release
/sig scheduling
Since has been marked resolved (fix available in 1.17), does that mean this issue is expected to go away in 1.17? Could you please give a try?
, fix for just resolved the case add event may be lost after deletion event. It doesn't solve the problem that a delete followed by an add may be delivered as one update notification and it's thought normal from MikeSpreitzer's comment. The main cause of this problem is scheduler cache use Pod UID as key while sharedInformer use namespace/name. Still, I also test with the latest kubernetes code. The bug of this issue reappeared.
Thanks I a comment in the PR, and a test is necessary to reproduce the issue and verify the fix.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale ping there is an unresolved comment in the PR, do you have a chance to take a look?


Question: timeOfLastSync:                  time.Time{}, volumesFailedReconstruction:     make([]podVolume, 0), volumesNeedUpdateFromNodeStatus: make([]v1.UniqueVolumeName, 0), <del> volumesNeedReportedInUse:        make([]v1.UniqueVolumeName, 0), </del> } }

remove and was introduced in to fix This is to overcome the situation that volume is to DSW at first and then deleted before it is to ASW, which causes mounted volume neither in DSW nor ASW and won't execute unmount/detach: The reason the volume cannot be to ASW in time is , which is . However, there is another PR also solved the issue. What it did is to check whether ASW already has the volume before deleting it from DSW, this makes sure that volume can be detached/unmounted after it is deleted from DSW. I suggest that we can keep the second solution and remove related code to make the reconstruction process and reconcile loop cleaner?
/sig storage
cc
Should I proceed with this issue? Your guidance is appreciated.
I think we'd better wait for the feedback from
cc
Sorry for a late reply. This is how behind we are with reviews :-(. Reading Now that is on by default, it reconstructs volumes first and always puts them to ASW, it should never happen that a reconstructed volume is in DSW and not in ASW, right? So we can remove easily. I would not be so sure without , because the code is 5 years old and who knows all the implications. You could be right though, it may not be necessary there too.
/triage accepted
/assign


Question: return claims } <ins> func claimWithDataSource(name, kind, apiGroup string, claims []*v1.PersistentVolumeClaim) []*v1.PersistentVolumeClaim { claims[0].Spec.DataSource = &v1.TypedLocalObjectReference{ Name:     name, Kind:     kind, APIGroup: &apiGroup, } return claims } </ins> func annotateClaim(claim *v1.PersistentVolumeClaim, ann map[string]string) *v1.PersistentVolumeClaim { if claim.Annotations == nil { claim.Annotations = map[string]string{}

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In clusters that have intree drivers as the default storageclass, if you try to create a PVC with snapshot data source and forget to put the csi storageclass in it, then an empty PVC will be provisioned using the default storageclass. What you expected to happen: PVC creation should not proceed and instead have an event with an incompatible error, similar to how we check proper csi driver in the csi provisioner. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
I think similar issue exists with cloning. We'll need to think about what the behavior should be for data populators too. cc
/triage accepted
Scenario to reproduce the issue Create the nginx deployment and the PVC used by it Exec into the running container, cd into the PVC mounted path and create some files Create the following using as source the The used is the following and wait until it becomes Delete the deployment and the initial PVC Create a new PVC, using the previously created as its Set the name of the new (restored) PVC to the nginx deployment and create the again into the PVC mounted directory. It should contain the previously created files but its empty
can you modify the title of this issue and change to ? In addition to snapshot datasource, using PVC datasource and AnyVolumeDataSource should also fail for in-tree PVC.
/assign
I wasn't 100% sure on data populators. Because it shouldn't have a dependency on CSI right?
I guess even if data populators does support intree volumes, pv controller should ignore it similar to csi provisioner
It may or may not have a dependency on CSI. Design is still in progress. We can leave it out for now until we know exactly how to use it. One thing I know is that the any volume data source cannot be an in-tree k8s resource.
Yes, we can actually just ignore it in pv controller.

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm getting same with longhorn, is this issue resolved on your side?


Question: name:              "valid Register", registry:          Registry{}, nameToRegister:    "pluginFactory1", <del> factoryToRegister: NewMockNoopPluginFactory(), </del> <ins> factoryToRegister: m1, </ins> expected: Registry{ <del> "pluginFactory1": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory1": m1, </ins> }, shouldError: false, }, { name: "Register duplicate factories", registry: Registry{ <del> "pluginFactory1": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory1": m1, </ins> }, nameToRegister:    "pluginFactory1", <del> factoryToRegister: NewMockNoopPluginFactory(), </del> <ins> factoryToRegister: m2, </ins> expected: Registry{ <del> "pluginFactory1": NewMockNoopPluginFactory(), </del> <ins> "pluginFactory1": m1, </ins> }, shouldError: true, },

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Below are the failures while running unit test with golang in master branch. Reference : $ go version go version devel go1.17- Thu Apr 22 01:40:02 2021 +0000 linux/ppc64le Note: Occurs in x86 architecture as well. The tests should Pass as with stable go git clone cd kubernetes make test KUBERACE=-race WHAT=./pkg/scheduler/framework/runtime/ Master branch of kubernetes repo. OS: Linux Observed failure on ppc64le and x86 architectures Below is the attached log:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I request help on getting this triaged. Is there a procedure to follow to get this accepted as a bug?
The Go compiler developer has identified the bugs in the test that is causing it to fail. They are listed in the last comment in the Go issue before closing it.
So the correct way to compare function would be something like would you test it in your env?
The issue still exists. Can you please reopen?
I am not sure if you can access this link for our CI job. Can you please check? https://prow.ppc64le- The above tests have been failing every day in our jobs!
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
downloads go from url="${GOLANGVERSION}.linux-" downloads go from "curl -X GET http://build-?project=build_commit -o " I suspect the postsubmit-master-golang-kubernetes-unit-test-ppc64le failed for wrong golang version.

failed for other problems. The TestMerge is not failed. For ppc64le failures, there are several issues in golang: (fixed yesterday)
Yes the go versions for both the jobs are different and it is intentional: : It is meant to run Kubernetes unit tests with the stable version of golang : It is meant to run Kubernetes unit tests with golang built from the master branch of As I mentioned in the description of this issue, these unit test cases are failing only on golang that is built from master branch and passing with stable versions of golang. And as Lynn mentioned in , the golang team has concluded that these failures are due to the bugs in the test case. And the golang issues that you mentioned do not seem to be related to the failures mentioned in this issue.
With , is false Per , do you mean that we should change to ?
No. In general comparing is the right thing to do. But if the Value is actually a , which is not comparable, comparing will panic.
/sig scheduling
Note from golang team about this bug is here:


Question: <ins> #!/bin/bash readonly reset=$(tput sgr0) readonly   red=$(tput bold; tput setaf 1) readonly green=$(tput bold; tput setaf 2) KUBE_HOOKS_DIR="$(dirname "$(test -L "$0" && echo "$(dirname $0)/$(readlink "$0")" || echo "$0")")" exit_code=0 echo -ne "Checking for files that need gofmt... " files_need_gofmt=() files=($(git diff --cached --name-only --diff-filter ACM | grep ".go" | grep -v -e "third_party" -e "Godeps")) for file in "${files[@]}"; do # Check for files that fail gofmt. diff="$(git show ":${file}" | gofmt -s -d)" if [[ -n "$diff" ]]; then files_need_gofmt+=("${file}") fi done if [[ "${#files_need_gofmt[@]}" -ne 0 ]]; then echo "${red}ERROR!" echo "Some files have not been gofmt'd.  To fix these errors, " echo "cut and paste the following:" echo "  gofmt -s -w ${files_need_gofmt[@]}" exit_code=1 else echo "${green}OK" fi echo "${reset}" echo -ne "Checking for files that need boilerplate... " files_need_boilerplate=() boiler="${KUBE_HOOKS_DIR}/boilerplate.py" # Check for go files without the required boilerplate. if [[ ${#files[@]} -gt 0 ]]; then files_need_boilerplate+=($("${boiler}" "go" "${files[@]}")) fi # Check for sh files without the required boilerplate. files=($(git diff --cached --name-only --diff-filter ACM | grep ".sh" | grep -v -e "third_party" -e "Godeps")) if [[ ${#files[@]} -gt 0 ]]; then files_need_boilerplate+=($("${boiler}" "sh" "${files[@]}")) fi # Check for py files without the required boilerplate. files=($(git diff --cached --name-only --diff-filter ACM | grep ".py" | grep -v -e "third_party" -e "Godeps")) if [[ ${#files} -gt 0 ]]; then files_need_boilerplate+=($("${boiler}" "py" "${files[@]}")) fi if [[ "${#files_need_boilerplate[@]}" -ne 0 ]]; then echo "${red}ERROR!" echo "Some files are missing the required boilerplate header" echo "from hooks/boilerplate.txt:" for file in "${files_need_boilerplate[@]}"; do echo "  ${file}" done exit_code=1 else echo "${green}OK" fi echo "${reset}" echo -ne "Checking for API descriptions... " files_need_description=() # Check API schema definitions for field descriptions for file in $(git diff --cached --name-only --diff-filter ACM | egrep "pkg/api/v.[^/]*/types.go" | grep -v "third_party"); do # Check for files with fields without description tags descriptionless=$("${KUBE_HOOKS_DIR}/description.sh" "${file}") if [[ "$descriptionless" -eq "0" ]]; then files_need_description+=("${file}") fi done if [[ "${#files_need_description[@]}" -ne 0 ]]; then echo "${red}ERROR!" echo "Some API files are missing the required field descriptions." echo "Add description tags to all non-inline fields in the following files:" for file in "${files_need_description[@]}"; do echo "  ${file}" done exit_code=1 else echo "${green}OK" fi echo "${reset}" echo -ne "Checking for docs that need updating... " if ! hack/verify-gendocs.sh > /dev/null; then echo "${red}ERROR!" echo "Some docs are out of sync between CLI and markdown." echo "To regenerate docs, run:" echo "  hack/run-gendocs.sh > docs/kubectl.md" exit_code=1 else echo "${green}OK" fi echo "${reset}" exit $exit_code </ins>

Multiple, for example: Since See above The test tolerations are now to strict: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think it's from if you check the cri-o versions the succeessful one was and the failing was
Working on a fix in
cc
/reopen need to bump in kube as well to fully get the fix
Reopened this issue. <detailsIn response to : /reopen </details


Question: // and check that the pod sees expected data, e.g. from the server pod. // Multiple Tests can be specified to mount multiple volumes to a single // pod. <ins> // Timeout for dynamic provisioning (if "WaitForFirstConsumer" is set && provided PVC is not bound yet), // pod creation, scheduling and complete pod startup (incl. volume attach & mount) is pod.podStartTimeout. // It should be used for cases where "regular" dynamic provisioning of an empty volume is requested. </ins> func TestVolumeClient(f *framework.Framework, config TestConfig, fsGroup *int64, fsType string, tests []Test) { <del> clientPod, err := runVolumeTesterPod(f.ClientSet, config, "client", false, fsGroup, tests) </del> <ins> testVolumeClient(f, config, fsGroup, fsType, tests, false) } // TestVolumeClientSlow is the same as TestVolumeClient except for its timeout. // Timeout for dynamic provisioning (if "WaitForFirstConsumer" is set && provided PVC is not bound yet), // pod creation, scheduling and complete pod startup (incl. volume attach & mount) is pod.slowPodStartTimeout. // It should be used for cases where "special" dynamic provisioning is requested, such as volume cloning // or snapshot restore. func TestVolumeClientSlow(f *framework.Framework, config TestConfig, fsGroup *int64, fsType string, tests []Test) { testVolumeClient(f, config, fsGroup, fsType, tests, true) } func testVolumeClient(f *framework.Framework, config TestConfig, fsGroup *int64, fsType string, tests []Test, slow bool) { clientPod, err := runVolumeTesterPod(f.ClientSet, config, "client", false, fsGroup, tests, slow) </ins> if err != nil { framework.Failf("Failed to create client pod: %v", err) }

Which jobs are flaking: gce-cos-master-alpha-features Which test(s) are flaking: [sig-storage] CSI Volumes [Driver: ][Serial] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with snapshot data source [Feature:VolumeSnapshotDataSource] Testgrid link: Reason for failure: It takes 6 minutes to start a pod with PVC that has been restored from a snapshot, while the test waits only 5 minutes: csi-attacher sidecar says: Perhaps the provisioner should wait until the volume is ready to be used? That may not help with this test timeout, as PVC is provisioned after the Pod is scheduled and the total time could be the same. All this is caused by my PR to test block snapshots, It reworked the tests from using . If snapshots are slow, should all persistent storage tests use / ?
/sig storage
Generally, cutting a snapshot is quick, but uploading can take longer. Does provisioner wait until the snapshot is "ready" before returning success?
cc
(But yes, we should increase the timeout because overall time will still be longer no matter which order we take)
The provisioner does not wait. If snapshot is not readyToUse, creating a volume from it will fail. I'll have to check to see if there's logic to wait in the e2e test. Regarding GCE PD, at restore time, do we need to download the snapshot from object store? If that's the case, it could take a while to download as well.
volume controller should keep trying until snapshot is ready. Looks like attacher tried to attach while disk is not ready. It is strange because I think attach only happen after volume is successfully provisioned.
AFAIK, most cloud providers' snapshot ATM will be uploaded to some remote when taking, and will be downloaded when restoring. GCE PD is one of them.
It's not taking snapshots what slows the test, it's restoring it - returns quickly, but the returned volume is not really usable and attach fails for ~4 minutes until the snapshot is fully restored somewhere in GCE. Now that I am thinking about it, maybe it's better if the driver returns the volume early, so it does not end up orphaned when user deletes corresponding PVC and the provisioner is restarted at the same time.

Anyway, I'll extend the timeout in volume checks. /assign
Which brings me to idea: what about other storage backends? Some of them may take even longer that 10 minutes to restore a snapshot. Or more than 5 minutes to provision an empty volume and run a pod. There should be some configuration of timeouts (timeout multiplier?) for
I opened to investigate from GCE side if this behavior is expected.


Question: // The computation is done here to ensure the pod status used for it contains // information about the container end states (including exit codes) - when // SyncTerminatedPod is called the containers may already be removed. <del> apiPodStatus = kl.generateAPIPodStatus(pod, podStatus, true) </del> <ins> apiPodStatus = kl.generateAPIPodStatus(pod, stoppedPodStatus, true) </ins> kl.statusManager.SetPodStatus(pod, apiPodStatus) // we have successfully stopped all containers, the pod is terminating, our status is "done"

My pod completes (success or failure). The IP temporarily is removed on one of the status updates. The IP is persistent . Apply: Watch: The initial 'succeeded' should have the IP still. On older versions: Full pod state, from 1.27: <details</detailsPrior discussion many, see above Tested on kind Linux <details</details<details</details<details</details
/sig network maybe Relevant PRs in 1.27:
cc
/priority important-soon
is about endpointslices, so no related seems related , ping
I can confirm this commit specifically is the culprit (when I build and test the parent commit it works ok): For now I don't know the reason, cc in case you have some ideas. EDIT: when commenting out these it works as before. I suspect this is because the does not have the information which is determined in
I've opened a PR with a fix. PTAL: Additionally, it turns out that the bug causes one more unnecessary request with status update.
This also relates to sig-node: /sig node We talked about this in the SIG Network meeting, looks like you are taking a look: /assign Please let us know what you find, and please let us know if you need any support!
Thank you! I think I have got to the bottom and summarized the findings in the PR description: I would love to move forward with the review :)
the question from SIG Node meeting was - what is the impact of it besides the flickering IP on status. This may help justify the cherry-pick.
do you have some additional insights about the impact?
In our usage we had to handle events on a pod informer differently to account for the fact we may read it in an invalid state if we read at the wrong time. So for us technically it hsa no impact since we worked around it, for others it could be quite severe potentionally
Thanks! Just FYI as demonstrated by experiments on you could also face this situation in case of kubelet restart. So, probably good you put this workaround in place anyway.
/triage accepted People affected by this regression likely will be affected by kubelet restart in a very precise moment. But it is a low probability event so I would advocate to fix the regression by backporting this fix instead of forcing consumers to work around this behavior.


Question: <-stopCh } <ins> // addPodEvictorForNewZone checks if new zone appeared, and if so add new evictor. func (nc *NodeController) addPodEvictorForNewZone(node *v1.Node) { zone := utilnode.GetZoneKey(node) if _, found := nc.zoneStates[zone]; !found { nc.zoneStates[zone] = stateInitial if !nc.useTaintBasedEvictions { nc.zonePodEvictor[zone] = NewRateLimitedTimedQueue( flowcontrol.NewTokenBucketRateLimiter(nc.evictionLimiterQPS, evictionRateLimiterBurst)) } else { nc.zoneNotReadyOrUnreachableTainer[zone] = NewRateLimitedTimedQueue( flowcontrol.NewTokenBucketRateLimiter(nc.evictionLimiterQPS, evictionRateLimiterBurst)) } // Init the metric for the new zone. glog.Infof("Initializing eviction metric for zone: %v", zone) EvictionsNumber.WithLabelValues(zone).Add(0) } } </ins> // monitorNodeStatus verifies node status are constantly updated by kubelet, and if not, // post "NodeReady==ConditionUnknown". It also evicts all pods if node is not ready or // not reachable for a long period of time.

<!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T10:51:10Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.0-alpha.1.51+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-06-20T11:29:14Z", GoVersion:"go1.8", Compiler:"gc", Platform:"linux/amd64"} What happened: After adding label to a node, node controller panics. What you expected to happen: Everything should still work. How to reproduce it (as minimally and precisely as possible): Run and wait for the node controller to panic: Anything else we need to know: I've checked the code and it looks like the node controller expects the label to be present during node registration and it can not be later. Even if the label can not be latter it shouldn't panic.
/sig node
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign let me try this :).


Question: }) }) <del> ginkgo.It("should get and update a ReplicationController scale", func() { </del> <ins> /* Release: v1.26 Testname: Replication Controller, get and update ReplicationController scale Description: A ReplicationController is created which MUST succeed. It MUST succeed when reading the ReplicationController scale. When updating the ReplicationController scale it MUST succeed and the field MUST equal the new value. */ framework.ConformanceIt("should get and update a ReplicationController scale", func() { </ins> rcClient := f.ClientSet.CoreV1().ReplicationControllers(ns) rcName := "e2e-rc-" + utilrand.String(5) initialRCReplicaCount := int32(1)

[X] Test approval issue: [X] Test PR: 29 Sept 2022 [X] Two weeks soak end date: 14 Oct 2022 [X] Test promotion PR: According to this APIsnoop query, there are still two remaining ReplicationControllerScale endpoints which are untested for conformance. readCoreV1NamespacedReplicationControllerScalereplaceCoreV1NamespacedReplicationControllerScale Using a number of existing e2e test practices a new has been created for two ReplicationControllerScale endpoints. The e2e logs for this test are listed below. This query shows which scale endpoints are hit within a short period of running this e2e test If a test with these calls gets merged, test coverage will go up by 2 point This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: prometheus.MustRegister(shortWatchesTotal) prometheus.MustRegister(watchDuration) prometheus.MustRegister(itemsPerWatch) <ins> prometheus.MustRegister(lastResourceVersion) </ins> cache.SetReflectorMetricsProvider(prometheusMetricsProvider{}) }

I'm seeing a large number of unique prometheus series showing up (due to the logging we for anonymous reflectors). This is a small cluster but I believe this may have unbounded cardinality.
we storing the resource version in the metric name?
cc
Came into the same problem, checked the code, seems we use + + as metric names, for most reflector, they use a default name provided by which is go filename. I think the proper way is use as the metric name, and add a label, like
opened a pr to fix this, ptal thanks :)


Question: } } <del> func TestMakePodStatus(t *testing.T) { </del> <ins> func TestPodStatusWithBadNode(t *testing.T) { </ins> fakeClient := client.Fake{ MinionsList: api.NodeList{ Items: []api.Node{

I'm actually not sure if it's by design. But certainly replication controller doesn't behave like it expects this.
If restart policy is to restart always, they should be restarted, but in that case they shouldn't be reported as failed.
This is a known issue, and by design based on upper-level will observe the status reported from kubelet, but not check with RestartPolicy and generate PodCondition / PodStatus. When RestartPolicy is Always, even Kubelet reports all containers are at failed state, apiserver shouldn't mark Pod as failed state. This is why initially I want kubelet report Pod state / status / condiction.
As a quick fix, we could let apiserver populate the right state / status based on containerStatus and RestartPolicy for now. In a long term, I still think we should make kubelet smarter and atomic and self-contained component, not just a wrapper around docker.
Short term fix should be P1 then we can de-prio for a "realer" fix.
Yes, I am working on the short term fix, and should have it included in 0.6 cut.
are you still working on this?
Yes, I am working on this. Should finish it today.


Question: syncPeriod time.Duration, minSyncPeriod time.Duration, excludeCIDRs []string, <ins> strictARP bool, </ins> masqueradeAll bool, masqueradeBit int, clusterCIDR string,

/kind bug Kubernetes version: 1.13.1. kube-proxy mode: IPVS. SDN: flannel. We are used services like this: where IP 192.168.10.201 is from our internal /24 network (in particular, our Kube nodes have IPs 192.168.10.1, 192.168.10.2 etc). Note that there is no conflicts between real IPs and virtual external ones. Before updating from 1.12.3 to 1.13.1, it worked correctly. In partucular these IPs was accessible from other non-Kubernetes hosts from 192.168.10.0/24 network. After update kube-proxy to 1.13.1, this feature stops working: non-Kubernetes hosts cannot resolve service IPs via ARP: On Kubernetes hosts it still working, because not requires ARP resolving. If we reset sysctl parameter arpignore on Kube hosts, all works normally even from non-Kube hosts. Quck debugging shows that problem is in the commit . Please make this behaviour opt-out. (Now we resets arpignore to 0 via cron every 1 min.)
/sig network
Having the same problem on Kubernetes 1.13.2 btw Thanks for a workaround!
Same problem, fix this please!!!
This is similar to We've had a discussion there on how to best implement a solution but haven't decided yet I'm in favor of flag which defaults to the previous behavior to avoid breaking existing setups (alternatives are cleaner but involve creating an additional arptable rule per service which I think we should avoid) what do you think?
OK

What happened: After PR: , we noticed communications issue in our pods. We use the Lyft CNI plugin () which relies on two interfaces in pod: an IPVLAN interface connecting to the AWS VPC which has a VPC IP an Unnumbered point-to-point interface (veth) connecting to the host and that does not have an IP (it "borrows" the one from the IPVLAN interface) On the host side, there is a veth interface which also doesn't have an IP (it borrows one from the main interface). With the latest versions of kube-proxy communication between host and pods break What you expected to happen: Communication still works How to reproduce it (as minimally and precisely as possible): Use the Lyft CNI plugin Deploy kube-proxy Try to ping the host from a pod (or a pod from the host) Anything else we need to know?: The new sysctls create 2 issues: containers started after kube-proxy inherit so they will not answer ARP queries on the veth interface the veth on the host side also does not have an IP so it will no answer ARP queries from the pods I know that our setup is pretty uncommon but assuming that the veth on the host side will have an IP seems dangerous. Some discussions on the PTP plugin for instance suggest not using IP for the host interface: containernetworking/plugins Possible fixes: create a flag to enable ARP filtering (I think to avoid breaking existing setups, we should disable it by default but I'm fine with enabling it by default as long as we document the solution) use arptables to filter ARP queries for clusterIPs (): a bit more work because we need to do it for all clusterIPs (there may be a more efficient way to do it but I could not make it work by filtering on the kube-ipvs interface) and this requires the arptables binary. We could test for the existence of the binary and log a warning if it is missing (similar to behavior with conntrack) Environment: Kubernetes version (use ): kube-proxy OS (e.g. from /etc/os-release): Kernel (e.g. ): /area ipvs /sig network /kind bug /assign ping
I can say only that solution with arptables working fine with MetalLB.
Oh, really? Glad you confirm that.

But I would like to make sure that we won't create too many arptables rules. Can ipset works with arptables?
It doesn't look like arptables works with ipset
Unfortunately doesn't support ipsets, but new do. (not fully implemented yet) Eg we can create ipset for arp: But current implementation is not allowing to use it for arp rules for now: Will return: I guess because there is still no type specified for (destination ip) and (source ip) BTW: nft does not allow to create set of integer components: But there is a way for specify multiple IPs in one rule, eg: Where is IPv4 addresses in decimal format More info here:
Another option would be to use the and filter all queries but this flag is not in kube-proxy today
Bad idea, it will handle but not - which can be really differ
can you explain more detail why Lyft CNI can not assign IPs to the right interfaces for avoid this kind of issues?
Yes you are right. bad idea It could definitely add IPs to these interfaces but it's not the choice that was made. More details here: ("Unnumbered point-to-point interface"). I could probably PR a fix there. I'm a bit worried that some other CNI plugins could break here though
We are about to deploy some 1.13 clusters and can't use kube-proxy 1.13 as is We could start with a custom build but I assume others will be impacted The simplest way forward for now is probably to use a flag to control this behavior. I think the safest is probably to disable this by default to avoid the issue for people upgrading but if you think it is really important we can enable it by default. I know we'd like to avoid adding more flags but other solutions seem a bit more complicated and will most likely result in one iptables rule per service. What do you think ?
Just FYI, we have another CNI plugin that is affected by this.
We addressed the issue by assigning a link-local address 169.254.1.1/32 to the host-side veth in . So this issue is not critical at least for our case.
Good news everyone, new nft supports setting ip address for arp filters : that's makes possible for using sets


Question: <del> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package sets // ordered is a constraint that permits any ordered type: any type // that supports the operators < <= >= >. // If future releases of Go add new ordered types, // this constraint will be modified to include them. type ordered interface { integer | float | ~string } // integer is a constraint that permits any integer type. // If future releases of Go add new predeclared integer types, // this constraint will be modified to include them. type integer interface { signed | unsigned } // float is a constraint that permits any floating-point type. // If future releases of Go add new predeclared floating-point types, // this constraint will be modified to include them. type float interface { ~float32 | ~float64 } // signed is a constraint that permits any signed integer type. // If future releases of Go add new predeclared signed integer types, // this constraint will be modified to include them. type signed interface { ~int | ~int8 | ~int16 | ~int32 | ~int64 } // unsigned is a constraint that permits any unsigned integer type. // If future releases of Go add new predeclared unsigned integer types, // this constraint will be modified to include them. type unsigned interface { ~uint | ~uint8 | ~uint16 | ~uint32 | ~uint64 | ~uintptr } </del>

Package use a custom-defined type . Go 1.21 support an official package cmp which has .
/sig api-machinery
cc
/assign xgp01
/triage accepted
/kind cleanup


Question: "k8s.io/kubernetes/pkg/api" "k8s.io/kubernetes/pkg/api/errors" client "k8s.io/kubernetes/pkg/client/unversioned" <ins> "k8s.io/kubernetes/pkg/controller/endpoint" </ins> "k8s.io/kubernetes/pkg/labels" "k8s.io/kubernetes/pkg/types" "k8s.io/kubernetes/pkg/util"

This is mostly motivated by petset. Currently we create a governing service that is responsible for the network identity of each pet. The requirements of this governing service differ from a normal service in the following ways: have internal protocols to notice peer health, and a list of different checks (eg: ). In kube, peers use DNS to communicate. So we don't want a readiness check failure to remove the pet from DNS and abruptly cut it off from the set. "governing service" is headless so kube-proxy doesn't need to NAT the service ip. will not be using this Service, so it doesn't matter that unready pets show up in DNS under the svcName sub domain. In addition to these, there's one important reason some pets need to have DNS before readiness. To provide the guarantees required to safely bootstrap a pet, we need: 0 starts before pet1.. 't start pet1 till pet0 is running/ready pet1 starts, it sees pet0 in DNS The easiest way to guarantee 3 today is to do one of: poll in the entrypoint of pet0, till it sees itself poll in the init container of pet0, till it sees itself poll in the init containers of pet {1..N-1} till it sees all previous pets Third option is not a great idea because we don't know who else is supposed to exist, and for the first 2 to work with a readiness probe, we need a DNS record before the Service is ready.
I have a wip that does this through an annotation:
+1. this is useful even in traditional n rc's and n+1 services setup for clustered systems like n nodes zookeeper/etcd. Currently, there isn't a easy way to specify a readiness check in such setups, because the check in these cases would look for whether the pod member has joined the cluster successfully as follower, which wouldn't work as they discover each other through them appearing in service endpoints. this prevents one from being able to do a rolling upgrade of cluster members safely, because there is no readiness feedback after a member pod is upgraded.
Suggestion to hold this for alpha in 1.5, and in 1.6 spend a bit of time talking about how services are best intermixed with stateful sets (there's a lot of correlation between them).

That avoids us breaking people in 1.5 and gets a bit more think time.
SGTM
See also
Can we claim that if a pod is requesting pet DNS (i.e DNS name via the hostname/subdomain annotation) it never wants kube-style readiness, because it has its own internal protocol for peer health? If that's true we can get rid of the annotation and teach kubedns to always serve records for pet DNS ignoring readiness. My intuition tells me the answer to that question is no (i.e there exist pets that use DNS for discovery, and don't have an internal protocol for readiness, so they must use kube readiness to remove themselves from the peer group), and so the ability to toggle it on/off is useful, but I also want others to consider it.
I'd love to nix an annotation but I have not the expertise to answer this one.
I don't have a strong feel either way - I'd say readiness controls on a service feel orthogonal to independent names though. Since we don't have per port readiness, we need to consider that at some point we might need more sophistication on services or pods to control membership (unless we make label flipping easier). I would like to see us tease apart the "leader / follower" service member concept a bit more because that might inform how we handle readiness (seems like a separate bit, but we may want atomic changes).
They are somewhat orthogonal, but you presumably only want to discover healthy peers, unless you have a way to probe every peer you discover for health. Applying this policy based on endpoint info also feels a little weird, because it's harder to create overlapping Servics with different behavior. I think doing the common thing always and by default is fine, as long as we can annotate our way back to what we have today, but what form would the opposite of today's Service level annotation take? i.e something that says "even though my endpoints need pet names, apply the regular rediness behavior to their DNS records"
How bad would it be to start without an annotation, assume that pet-names == tolerate-unready, and see if anyone screams? Who should we consult?
SGTM, that's also what I was prposing in the previous comment
I guess the question is why do we even want to do that? Why make a change that couples logic vs leaving as is with decoupled logic?

We have people who want unready endpoints for deployments to form gossip pools - so I guess that counts as a use case where pet names is not coupled to unready. Is leaving the annotation on for another release really a problem? , Prashanth B <:
That's exactly the kind of use case I was looking for, something that doesn't require pet names. I get the feeling there will be other apps but I couldn't produce an example.
JBoss EAP uses it to form initial quorum for jgroups, which isn't a strong consensus system. So they use deployments and a minimum set to form an initial quorum and deal with split brain like jgroups normally does (infinispan has multiple consistent modes, some of which might prefer loose pools). Arguably smaller sets and some infinispan quorums might prefer statefulsets, but those use cases are also more tolerant of partitions in general. , Prashanth B <:
we are using tolerate-unready-endpoints annotation in our zookeeper deployments (not using pet/stateful set). Without unready endpoints, it is impossible to define a meaningful pod readiness check for zk, and without a readiness check, it is hard to do rolling updates (one ensemble member at a time).
The question is whether there's a need to have StatefulSets without tolerate-unready-endpoints. It sounds like maybe yes, which makes me grumpy, but I'll acquiesce if Clayton and Prashanth say so. , ravilr <:
isn't it enough to establish a need to tolerate-unready without statefulsets?
I have two services pointing to my etcd stateful set - one that is endpoints (peers) and tolerates unready. I have one that is for naive clients that uses readiness. On Dec 1, 2016, at 2:59 AM, Prashanth B <: isn't it enough to establish a need to tolerate-unready without statefulsets? — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
fine.
/sig network
Right. This is exactly why etcd operator uses the alpha feature. Can we make it beta in 1.8 release?
/cc
We had discussed promoting since we need it for StatefulSet and no one has come up with anything better.

We should make a decision and include it in our GA roadmap for workload controllers.
Be sure to include sig-network On Jul 6, 2017, at 2:40 PM, Anthony Yeh <: had discussed promoting tolerate-unready-endpoints since we need it for StatefulSet and no one has come up with anything better. We should make a decision and include it in our GA roadmap for workload controllers. — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
This is also useful for ingresses that need to retain the source IP using OnlyLocal node ports, which is important for firewalling with things like Calico. The problem that we see now is when a deployment occurs, the pod is removed from the node port before the PreStop completes, which doesn't give the upstream AWS load balancer time for the pod to fail out using normal health checks.
/close
Headless Services are a thing now


Question: // available volumes await a claim case api.VolumeAvailable: <ins> // TODO:  remove api.VolumePending phase altogether _, exists, err := volumeIndex.Get(volume) if err != nil { return err } if !exists { volumeIndex.Add(volume) } </ins> if volume.Spec.ClaimRef != nil { _, err := binderClient.GetPersistentVolumeClaim(volume.Spec.ClaimRef.Namespace, volume.Spec.ClaimRef.Name) if err == nil {

In case of cluster crash and status recovery, the persistentVolumeClaimBinder would not re-add available volumes to its internal index. Users would not be able to claim those volumes.
This is really bad. I think we should just nix Pending phase and always consider every Available volume for indexing - map lookups are cheap.


Question: glog.V(5).Infof("Modifying uninitialized resource %s", a.GetResource()) <ins> if updated != nil && len(updated.Pending) == 0 && updated.Result == nil { accessor.SetInitializers(nil) } </ins> // because we are called before validation, we need to ensure the update transition is valid. if errs := validation.ValidateInitializersUpdate(updated, existing, initializerFieldPath); len(errs) > 0 { return errors.NewInvalid(a.GetKind().GroupKind(), a.GetName(), errs)

There are two issues: doesn't have merge key The validation code regards it as an error if both Initializers.Result and Initializers.Pending are nil. So if multiple initializer controllers sending patches removing themselves from Initializers.Pending at the same time, the last patch that gets applied will get a validation error.
/sig api-machinery


Question: <ins> Copyright (c) 2013 Joshua Tacoma. All rights reserved. Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met: * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. * Neither the name of Google Inc. nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. </ins>

we have one in k/k and one in k/utils, both need to be in LICENSES /assign
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area code-organization
I think we should include anything matching the LICENSE/COPYRIGHT/COPYING patthers under or subdirs in k/k or dependencies... a quick sweep showed these:
/sig release ?


Question: echo "Using master: $KUBE_MASTER (external IP: $KUBE_MASTER_IP)" } <ins> # Reads kube-env metadata from master # # Assumed vars: #   KUBE_MASTER #   PROJECT #   ZONE function get-master-env() { # TODO(zmerlynn): Make this more reliable with retries. gcloud compute --project ${PROJECT} ssh --zone ${ZONE} ${KUBE_MASTER} --command  "curl --fail --silent -H 'Metadata-Flavor: Google'  'http://metadata/computeMetadata/v1/instance/attributes/kube-env'" 2>/dev/null } </ins> # Robustly try to create a static ip. # $1: The name of the ip to create # $2: The name of the region to create the ip in.

When we add nodes to an existing cluster, we need to reuse the existing configuration. In particular, we need to extract the kubelet cert & key, otherwise we get a cert error connecting to the master.
WIP fix in


Question: execAffinityTestForNonLBServiceWithTransition(f, cs, svc) }) <del> // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws") svc := getServeHostnameService("affinity-lb-esipp") svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal execAffinityTestForLBService(f, cs, svc) }) // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP on [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws") svc := getServeHostnameService("affinity-lb-esipp-transition") svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeLocal execAffinityTestForLBServiceWithTransition(f, cs, svc) }) // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should have session affinity work for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws") svc := getServeHostnameService("affinity-lb") svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeCluster execAffinityTestForLBService(f, cs, svc) }) // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. // [LinuxOnly]: Windows does not support session affinity. ginkgo.It("should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters] [LinuxOnly]", func() { // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. e2eskipper.SkipIfProviderIs("aws") svc := getServeHostnameService("affinity-lb-transition") svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeCluster execAffinityTestForLBServiceWithTransition(f, cs, svc) }) </del> ginkgo.It("should implement service.kubernetes.io/service-proxy-name", func() { ns := f.Namespace.Name numPods, servicePort := 3, defaultServeHostnameServicePort

Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: Services should be able to change the type and ports of a service [Slow] Testgrid link: https://k8s- Reason for failure: Test timeouts in ~half of the runs since it was enabled here: /sig scalability /sig network
/cc /cc
The context is - the problems are strictly related to scale - 5k-node jobs are timing out. I very briefly skimmed through the test and didn't see anything obviously suspicious (apart from the fact there are gazillions long timeouts there, which alltogether sum up long hours). That said - it seems to me that there are effectively two separate tests inside it: one for TCP one for UDP They don't seem to share anything with each other. - would it make sense to split this test two effectively two separate tests: one for TCP and one for UDP. (I guess it would also allow us to share some code there).
[And potentially it could help with debugging the problem if it is strictly related to only one of those cases...]
I wanted to focus on this past week but I didn´t have time, this is my list of more expensive times and approx average duration, all of the sig-network ones use LoadBalancers. I wanted to time the test to understand if there are places where can save time ... I don´t have much context on those tests, maybe Tim can help us more, but this comment makes me think that the use of both protocols is intended
/triage accepted
/triage accepted It's been a very long time since I touched this, and I recall holding my nose on it in the first place. Splitting TCP from UDP seems reasonable if we can run them in parallel. we'll want to share the flow, and parameterize some shared funcs. I don't know that this will prevent test timeouts, though.
We could use a volunteer to go poke at this, and see how hard it will really be.
You mean "forcing them to run in parallel"? Or "allowing them to run in parallel"? The later doesn't require any additiona work. The former I think mean that it will still have to be a single test (just the code will be organized better) and that won't really help with debugging (nor preventing timeouts).

this was discussed yesterday in sig-net meeting and my understanding was to split current test in 2 different so the test can run in parallel, if you want to do so and one protocol doesn´t have to wait for the other. I think that is what you meant here
As long as we CAN parallelize. Creating LBs can be sloooooow. :P Antonio Ojea <:
Reopening - I don't think just splitting will address the problem. It may help us with debugging though...
With the split we moved from 60 mins to run the test, to run 2 tests in [30-40] mins each https://k8s- Seems a win :smile:
Awesome - that's something I was silently counting for TBH :) [I doubt it will fix flakiness, but maybe it will make it easier to debug...]
Actually - I think we're getting close thanks to the two changes ( + this one). Here is the run: This particular one filed on UDP (not sure if it's just UDP-related, or just manifested with UDP now). And we have a clear failure now: In my PR I decreased the timeout from 2h to 45m, as it seemed to me that in the past it should be enough. But it failed on 45m timeout here. So I think the next question now is: is it that slow - that we need more than 45m or something isn't working as expected (either in the k8s itself or in the test) But I think we have a clear single place to look at now.
Actually - the full stacktrace: So it's this line that failed: What is interesting is that something was changing during those 45 minutes:
Note that it's 4th time we actually wait for LB in this test. The failure happens after the pod were scaled down to 0 and then to 1. So that makes me think, that's it's actually not LB-related, but rather something internally in k8s, becuase L4 LB doesn't depend on pods - it targets nodes...
If it is always UDP, I start to see ghosts between GCE and UDP ... we only have KIND to compare against, or do we have some other non gce enviroment?

I mean, there are other tests failing in UDP , and show much more failures in GCE than in kind
- this does not look "normal". Can we give some prio to helping track this flake?
hmm interesting. we will investigate.
/assign
- any progress on this?
I haven't had a chance to look into this yet. I will prioritize it this week.
Digging through the logs I believe this was a bug in the test code that was fixed by . In the test, the service port changes around 12:33. However due to the bug, the test continues to use the old service port as seen in the test output: The port should be 81. I believe the following sequence occurs from this point: 1) The reachability test fails in the beginning because the service has changed the port and the load balancer has not been updated yet resulting in the error. 1) Around 12:38, the service controller is syncing the service and notices that the service port has changed and starts to delete the forwarding rule causing the error to change to . 1) Once the forwarding rule is recreated, the error goes back to being because the forwarding rule exists but the wrong port is used. Looking at , the test runs after the one linked have all passed. Is there anything else that needs to be investigated as part of this issue?
OK - the test was quite flaky in the past but indeed it didn't flake for the last 3 weeks (neither this nor TCP one). I'm fine with closing this one - we can open a new one if this happens again. - thanks for debugging. - it seems that the splitting of the test actually made things waay better. Which is great.


Question: }) /* <del> Testname: var-expansion-subpath-ready-from-failed-state Description: Verify that a failing subpath expansion can be modified during the lifecycle of a container. </del> <ins> Release : v1.19 Testname: VolumeSubpathEnvExpansion, subpath ready from failed state Description: Verify that a failing subpath expansion can be modified during the lifecycle of a container. </ins> */ <del> ginkgo.It("should verify that a failing subpath expansion can be modified during the lifecycle of a container [sig-storage][Slow]", func() { </del> <ins> framework.ConformanceIt("should verify that a failing subpath expansion can be modified during the lifecycle of a container [sig-storage][Slow]", func() { </ins> podName := "var-expansion-" + string(uuid.NewUUID()) containerName := "dapi-container"

Is this a BUG REPORT or FEATURE REQUEST?: /kind api-change /sig storage /sig node /sig testing Description: Promote e2e tests for VolumeSubpathEnvExpansion to Conformance as it is GA approved. Related Issues: GA Upgrade: Feature Request: KEP:
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: obj := ctrl.claims.List()[0] claim := obj.(*v1.PersistentVolumeClaim) reactor.DeleteClaimEvent(claim) <del> for len(ctrl.claims.ListKeys()) > 0 { time.Sleep(10 * time.Millisecond) </del> <ins> err := wait.Poll(10*time.Millisecond, wait.ForeverTestTimeout, func() (bool, error) { return len(ctrl.claims.ListKeys()) == 0, nil }) if err != nil { return err </ins> } // claim has been removed from controller's cache, generate a volume deleted event volume := ctrl.volumes.store.List()[0].(*v1.PersistentVolume)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are some places in that wait for conditions . We should update all the places to set something like /help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! --What happened: There are some places in that wait for conditions . We should update all the places to set something like /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind cleanup /remove-kind bug
I can try to fix this :) /assign
/sig storage


Question: ResourceTypeOrNameArgs(args...). SingleResourceType(). Do() <ins> checkErr(r.Err()) </ins> mapping, err := r.ResourceMapping() checkErr(err)

I've reported this before.
From a Makefile I get exit status 255.
Just to be clear (sorry for the basic question), are you saying that when the service file is malformed the crash is unhelpful for debugging? Or that creating any/all services using is currently broken?
The YAML file is not malformed. I think there was something perhaps not quite right about firewalls/rules etc. and this caused to crash unhelpfully.