Question: defer resp.Body.Close() if resp.StatusCode != http.StatusOK { <del> klog.V(2).Infof("body of failing http response: %v", resp.Body) </del> <ins> klog.V(2).InfoS("Failed to read URL", "statusCode", resp.StatusCode, "URL", url) </ins> return nil, &HTTPError{ StatusCode: resp.StatusCode, URL:        url,

Log messages like are not very informative. /sig cloud-provider Either actual response body or none of it if this is a security concern. /sig security sig security to confirm it is ok to change this to log an actual response body. two places it is being logged: version (use ):1.20, but doesn't matter Cloud provider or hardware configuration:GKE OS (e.g: ):COS
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI, already handle []byte display in , may be we can move this to structured-logging.
/assign
Hi, I just found a place to test my first submit. If you are still working on this, please let me know. I'll close my PR
Hi, l still working on this issues, thanks .
OK, go ahead. This PR can be used as a reference, I‘ll close it
/assign


Question: // Don't call StartRecordingToSink, as we don't need neither refreshing event // series nor finishing them in this tests and additional events updated would // race with our expected ones. <del> broadcaster.startRecordingEvents(stopCh) </del> <ins> err = broadcaster.startRecordingEvents(stopCh) if err != nil { t.Fatal(err) } </ins> recorder.Eventf(regarding, related, isomorphicEvent.Type, isomorphicEvent.Reason, isomorphicEvent.Action, isomorphicEvent.Note, []interface{}{1}) // read from the chan as this was needed only to populate the cache <-createEvent

Follow-up cleanup work for: Most of the information about this issue can be found in , where the work merged with a to change the function signature of to return a func() as well as an error. This issue will help to track that. All TODOs introduced in are resolved. N/A N/A <detailsN/A </details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /priority low Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc /triage accepted
/priority important-longterm
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // and skips if a disk of that type does not exist on the node func SkipUnlessLocalSSDExists(config *localTestConfig, ssdInterface, filesystemType string, node *v1.Node) { ssdCmd := fmt.Sprintf("ls -1 /mnt/disks/by-uuid/google-local-ssds-%s-%s/ | wc -l", ssdInterface, filesystemType) <del> res, err := config.hostExec.IssueCommandWithResult(ssdCmd, node) </del> <ins> res, err := config.hostExec.Execute(ssdCmd, node) utils.LogResult(res) </ins> framework.ExpectNoError(err) <del> num, err := strconv.Atoi(strings.TrimSpace(res)) </del> <ins> num, err := strconv.Atoi(strings.TrimSpace(res.Stdout)) </ins> framework.ExpectNoError(err) if num < 1 { framework.Skipf("Requires at least 1 %s %s localSSD ", ssdInterface, filesystemType)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : separate stdout and stderr command exit code: remove kubectl dependency: Why is this needed: to replace
/assign
/sig storage HostExec was originally written for local PV tests. If we want to replace SSH in e2e, some features must be supported and I’d like to help on this. cc


Question: * [kubectl config](kubectl_config.md)	 - config modifies kubeconfig files <del> ###### Auto generated by spf13/cobra on 8-Dec-2015 </del> <ins> ###### Auto generated by spf13/cobra on 8-Jan-2016 </ins> <!-- BEGIN MUNGE: GENERATED_ANALYTICS --> [![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/kubectl/kubectl_config_set-cluster.md?pixel)]()

It acts like a boolean flag but requires an argument, unlike boolean flag is pretty much every environment. We need to find a better way to express the things that this is expressing. Maybe a add a tristate to the pflag lib that behaves the same way but doesn't demand an argument?
cc
The handles "true", "false", "unset" and has a default value. I seem to recall you were looking at changing it, but we needed to preserve the value. The complain here is that you have to do instead of just . /cc
fixing is really easy....


Question: reconcileLong = templates.LongDesc(` Reconciles rules for RBAC Role, RoleBinding, ClusterRole, and ClusterRole binding objects. <del> This is preferred to 'apply' for RBAC resources so that proper rule coverage checks are done.`) </del> <ins> Missing objects are created, and the containing namespace is created for namespaced objects, if required. Existing roles are updated to include the permissions in the input objects, and remove extra permissions if --remove-extra-permissions is specified. Existing bindings are updated to include the subjects in the input objects, and remove extra subjects if --remove-extra-subjects is specified. This is preferred to 'apply' for RBAC resources so that semantically-aware merging of rules and subjects is done.`) </ins> reconcileExample = templates.Examples(` # Reconcile rbac resources from a file

What happened: I noticed some surprising behavior of , which I'm not sure is intentional, but is very unexpected, in that will auto-create namespaces if it doesn't exist. This behavior does not seem to be documented anywhere. For example, if I run: against the following Role: Then the namespace gets created. This is confirmed when enabling logs in kubectl: What you expected to happen: is described in the CLI description as "preferred to 'apply' for RBAC resources so that proper rule coverage checks are done." However, this cannot be used as a drop-in replacement for apply as it requires additional privileges on the client, and the behavior is not symmetrical when the namespaces do not exist. I would expect the command to fail if the namespace does not exist. In the case of you simply get NotFound Environment: Kubernetes version (use ): /sig cli /sig auth <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
It seems that this is a intended feature
The code that backs is used on the server side as well. Hence the namespace creation was in . I made it best effort in which will land in 1.13.
Changing it to best effort doesn't actually help our specific situation because the deployment tool we are building (Argo CD), relies on running the kubectl CLI to perform deployments, and has root access. After changing the command from: to: Suddenly we were creating namespaces in some situations when we didn't expect it to be created, whereas before we could rely on an error coming back from about the missing namespace. Since Argo CD runs as root, best effort doesn't help. The simple workaround on our end is to just preface it with a : But I would suggest documenting this behavior at least.
It is intentional. +1 for documenting.
/assign opened


Question: assert.Nil(t, err, "unexpected error in NewProxyServer, addr: %s", addr) assert.NotNil(t, proxyserver, "nil proxy server obj, addr: %s", addr) assert.NotNil(t, proxyserver.IptInterface, "nil iptables intf, addr: %s", addr) <ins> assert.True(t, proxyserver.CleanupAndExit, "false CleanupAndExit, addr: %s", addr) </ins> // Clean up config for next test case configz.Delete("componentconfig")

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: bug What happened: Got panic after running: Panic: What you expected to happen: No panic and exit successfully. How to reproduce it (as minimally and precisely as possible): Run the following command on both CoreOS v1298.7.0(bare metal) and Ubuntu 16.04.2 LTS(OpenStack) Happened with the following versions: v1.7.0-alpha.3 v1.7.0-alpha.4 v1.7.0-beta.0 v1.7.0-beta.1 v1.7.0-beta.2 v1.7.0-rc.1 v1.8.0-alpha.0 v1.8.0-alpha.1 Anything else we need to know?: I guess it caused by . Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Both CoreOS v1298.7.0(bare metal) and Ubuntu 16.04.2 LTS(OpenStack) Kernel (e.g. ): CoreOS v1298.7.0: Linux node- 4.9.16-coreos-r1 SMP Tue Apr 4 19:09:38 UTC 2017 x8664 Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz GenuineIntel GNU/Linux Ubuntu 16.04.2 LTS: Linux ubuntu- 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: Others:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig network /area kube-proxy
/cc
cc/ Looks like there is a new regression in v1.7. Please triage this to decide if it is a release blocker. If not, please provide the workaround for the users. Thanks!

cc/
Digging into the code, will send a PR.
PR sent. Folks review it please.
Added the release-blocker label. We got a PR for it very fast so it won't delay the release anyway, thanks


Question: deps = [ "//vendor:github.com/davecgh/go-spew/spew", "//vendor:github.com/ghodss/yaml", <ins> "//vendor:k8s.io/apimachinery/pkg/runtime", </ins> "//vendor:k8s.io/apimachinery/pkg/util/mergepatch", ], )

Extracted from The RawExtension type allows an API object to inline arbitrary JSON. strategicpatch.CreateTwoWayMergePatch returns an error trying to map that inlined JSON to struct fields. It should be able to compute a patch for an object containing a field of that type.
cc
We need to decide what strategic merge patch should do in the following scenarios: unknown fields (can be encountered when the API or a user-defined object has a new field the compiled-in struct doesn't have yet) unmergeable fields (like RawExtension) cc
The current behavior is: unknown fields: trying to merge a unknown field will fail when lookup the metadata in the struct tags unmergeable fields: will be replaced as raw byte array Maybe we can use generic json merge for unmergeable fields?
Did that change since 1.5? In 1.5, we were getting an error during patch computation for objects with a RawExtension field
Sorry. This behavior is in 1.4.7. e.g . It will error in 1.5.
Yeah, the field gets replaced. Even in 1.4.7, an unknown field (like initcontainers in pods) or an object including a RawExtension (anything with a field of type or ) would error
My personal favor would be to instruct the patch to ignore such field, seems the cleanest approach to me. Similarly to how we allow annotating deletion and merging.
wouldn't it be confusing to submit a patch with stuff in that field, have the patch accepted, and have none of your submitted changes take effect? seems like the least surprising thing would be to treat struct fields like or as
How about both, iow. having as a default and as an opt-in. This would allow the resource author to decide how his resource should be treated. Eventually on we could return the ignored fields. Having said that we could start with right away. I think I could work on a patch implementing that if majority agrees (
I think is a good place to start when computing/applying patches for known struct fields like RawExtension. I'll open a separate issue for client-side handling of computing patches for unknown fields
Ignore the field sounds like a really good solution. WDYT of printing a warning to stdeer?
what does ignoring instead of replacing get us?
Sorry I was reading the comment on my phone between other things. Ignore my comment.


Question: } func (c *cephFSDriver) CreateDriver() { <del> // Reset config. It might have been modified by a previous CreateVolume call. c.driverInfo.Config.ServerConfig = nil </del> } func (c *cephFSDriver) CleanupDriver() {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce/gke multizone/regional jobs Which test(s) are failing: Only PD tests Since when has it been failing: Around 12/21 with the 16:49 run Testgrid link: https://k8s- Reason for failure: TBD Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
The runs started flaking after merged. cc Here's one failing run: Pod pod-subpath-test-gcepd-dynamicpv-pfsm failed to schedule because of conflicting node selectors: We shouldn't need to set the node selector for PD tests using PVs.
The PR broke the subpath test because the modification to the config struct embedded in the driver info now persists across tests: A quick fix would be to reset that field in CreateDriver. The long-term fix is the change discussed in My preference is to do the quick fix now, then either merge or close PRs in this order: (needs a decision about logging of tests that aren't getting defined, see ) a future PR for What I'd like to avoid is having to rebase PR on top of the long-term solution for issue - that'll be lots of code conflicts.
/assign PR is here: PR


Question: } } } <del> delete(plugin.podCIDRs, id) glog.V(3).Infof("Calling cni plugins to remove container from network with cni runtime: %+v", rt) if err := plugin.cniConfig.DelNetwork(plugin.netConfig, rt); err != nil { return fmt.Errorf("Error removing container from network: %v", err) </del> <ins> if err = plugin.delContainerFromNetwork(id, rt); err != nil { return err </ins> } return nil

I gathered e2e test results of suite for last 100 runs. The following is the list of flaky tests. Some might not be related to . I will start with the service tests. Any help would be appreciated. Detail list: Jenkins project:
cc/ Cluster team is working on to make kubenet default. Once we switch this to production, we don't need to restart docker to configure cbr0.
After a bunch of PRs got merged, the kubenet suite is much more stable. One more known flake can be fixed by I will post new flake tests here.
Just bump into a new failure: One of the nodes () got panic: Looks like it crashed at shaper reset.: Can you take a look?


Question: } // Wrap EmptyDir, let it do the setup. <del> wrapped, err := b.plugin.host.NewWrapperMounter(b.volName, wrappedVolumeSpec, &b.pod, b.opts) </del> <ins> wrapped, err := b.plugin.host.NewWrapperMounter(b.volName, wrappedVolumeSpec(), &b.pod, b.opts) </ins> if err != nil { return err }

Data as of June 3rd, 9:00 PM Eastern. Latest data can be found here: Prow link to the 2 jobs: * Additional Context:
/sig storage cc
I'm working on , it's hard to get a job yaml right at the first time. I'll check iscsi. /assign
/triage accepted /kind failing-test
ci-kubernetes-e2e-storage-kind-disruptive fixed: has 1 green run pull-kubernetes-e2e-storage-kind-disruptive fixed: , it fails when it should fail, it's green otherwise, I'll continue with iscsi.
/reopen it's still red, but at least it installs now:
Reopened this issue. <detailsIn response to : /reopen it's still red, but at least it installs now: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
e2e-gce-iscsi-serial job passes now: e2e-gce-iscsi is getting greener: I'll let it run for few more days to see if today's failures are flakes. /reopen
Reopened this issue. <detailsIn response to : It's getting greener: /reopen </details
iscsi-serial is green:


Question: switch label { case "name": fallthrough <del> case "Status.Phase": </del> <ins> case "status.phase": </ins> fallthrough <del> case "Status.Host": </del> <ins> case "status.host": </ins> return label, value, nil default: return "", "", fmt.Errorf("field label not supported: %s", label)

We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.


Question: kube::build::run_build_command make test-integration fi <ins> kube::build::copy_output </ins> if [[ "${FEDERATION:-}" == "true" ]];then ( source "${KUBE_ROOT}/build/util.sh"

It fails with: cc
Looking at the generated I see that FEDERATIONIMAGETAG is "" which is resulting in this error. I see that before , FEDERATIONIMAGETAG was being set to but that docker tag file does not exist for hyperkube. I see the following exist in my : but does not exist there. Trying to figure out why.
cc
make sure you are running the scripts with env var exported. That is what controls whether gets created.
With , we are not generating the federation-apiserver binary and instead using the hyperkube binary.
sorry, i copied the wrong line. The droid you're looking for: File conditionally written during process File consumed when templating federation control plane yaml
Right. My is also empty. I am trying to run federation e2e tests. The command I am running is: which is what I have always used. I guess something changed in
that should definitely work. Please and then run the step, capture the output and email to me? I was literally doing exactly this process for another PR today, and I verified the image tag file was getting written correctly as well.
I figured out this is happening because after make clean or a fresh clone, kubectl will not be available when we are trying to get the semanticimagetag_version() in here.
Awesome! Thanks


Question: package disk import ( <ins> "bytes" "crypto/sha256" "fmt" </ins> "net/http" "os" "path/filepath" "github.com/gregjones/httpcache" <del> "github.com/gregjones/httpcache/diskcache" </del> "github.com/peterbourgon/diskv" "k8s.io/klog/v2" )

We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests).

Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package.

I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e.

not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.


Question: } func TestWithMaxWaitRateLimiter(t *testing.T) { <del> limiter := NewWithMaxWaitRateLimiter(DefaultControllerRateLimiter(), 500*time.Second) </del> <ins> limiter := NewWithMaxWaitRateLimiter(NewStepRateLimiter(5*time.Millisecond, 1000*time.Second, 100), 500*time.Second) </ins> for i := 0; i < 100; i++ { if e, a := 5*time.Millisecond, limiter.When(i); e != a { <del> t.Errorf("expected %v, got %v", e, a) </del> <ins> t.Errorf("expected %v, got %v ", e, a) </ins> } } <del> for i := 100; i < 5100; i++ { if e, a := 500*time.Second, limiter.When(i); e < a { </del> <ins> for i := 100; i < 200; i++ { if e, a := 500*time.Second, limiter.When(i); e != a { </ins> t.Errorf("expected %v, got %v", e, a) } } <ins> } </ins> <del> for i := 5100; i < 5200; i++ { if e, a := 500*time.Second, limiter.When(i); e != a { t.Errorf("expected %v, got %v", e, a) } </del> <ins> var _ RateLimiter = &StepRateLimiter{} func NewStepRateLimiter(baseDelay time.Duration, maxDelay time.Duration, threshold int) RateLimiter { return &StepRateLimiter{ baseDelay: baseDelay, maxDelay:  maxDelay, threshold: threshold, } } type StepRateLimiter struct { count     int threshold int baseDelay time.Duration maxDelay  time.Duration } func (r *StepRateLimiter) When(item interface{}) time.Duration { r.count += 1 if r.count <= r.threshold { return r.baseDelay </ins> } <ins> return r.maxDelay } func (r *StepRateLimiter) NumRequeues(item interface{}) int { return 0 } func (r *StepRateLimiter) Forget(item interface{}) { </ins> }

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --The unit test which was recently is flaking Sample failure error trace Full test report can be found The test is flaking because of the time mismatch in a condition and its observed that the while the control arrives some times its few milliseconds(8m19.997445973s) lesser than the expected time(8m20s) <!-- You can identify the SIG from the "prowjobconfigurl" on the testgrid dashboard for a test --
/assign
/sig api-machinery /kind flake A flake according to the job history: https://prow.ppc64le-
/cc who got the PR assigned /triage accepted


Question: if len(r.strValues) == 1 { buffer.WriteString(r.strValues[0]) } else { // only > 1 since == 0 prohibited by NewRequirement <del> buffer.WriteString(strings.Join(r.strValues, ",")) </del> <ins> // normalizes value order on output, without mutating the in-memory selector representation // also avoids normalization when it is not required, and ensures we do not mutate shared data buffer.WriteString(strings.Join(safeSort(r.strValues), ",")) </ins> } switch r.operator {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After creating a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values, pods get created on the appropriate nodes—and, fortunately, only on the appropriate nodes—but the pods get terminated and deleted quickly before they can start running. The cycle takes about ten seconds from pod creation through completion of deletion; the pod appears to be deleted within one second after its creation, but it takes about ten seconds for the deletion to complete. If I vary the DaemonSet's scheduling predicates, I find the following work to get pods running successfully: Use to choose a specific node. Use with an operator of "Exists" or "NotIn" to choose some nodes. Use with an operator of "In," but with only one value to choose some nodes. Use with a label that is not applied by the cloud provider. If I come up with my own label such as "special-hardware" and use kubectl label node to apply it, and use that label as a match expression key, it seems to work fine. It's the dynamically applied labels like "" and "" that trigger this problem. Without the above concessions, if I use with an operator of "In," but with more than one value, the pods don't start correctly. Note that this behavior is a regression from Kubernetes version 1.10.4. The same configuration works as intended in a cluster running that earlier version. What you expected to happen: After creating the DaemonSet, pods would start successfully on the nodes with label values matching one of the DaemonSet's candidates values. How to reproduce it (as minimally and precisely as possible): a DaemonSet that uses node affinity to request only creating pods on nodes with a label matching a set of candidate values. that at least one node has a label with a value that matches a member of that candidate set. kubectl get pods -o wide to observe that there are pods created on behalf of the DaemonSet on nodes that match the predicate, but they have status "Terminating." By watching the pods, you can see new pods arrive with status "Pending," then "ContainerCreating," then "Terminating," which they'll retain until deletion completes and a replacement arrives. a similar pod directly, without a supervising DaemonSet.

If the "" label is inconvenient within your cluster, instead consider using "" and a few hostnames as the candidate values. that one pod starts running successfully on one of the nodes that match that the predicate. that lone pod (for clarity), and vary the predicate on the DaemonSet to see which changes allow pods to start running successfully. Try each of these individually: Remove one of the values in the sequence. That is, leave only one candidate value. Change the operator to "NotIn," and adjust the values to select some subset of the nodes. Alternately, try "Exists" with no values. Remove the node affinity stanza's and add a node selector in its place: Anything else we need to know?: Here is the DaemonSet object as captured via kubectl get daemonset -o yaml: Here is one of the pods created on behalf of the DaemonSet: Here is one of the nodes on which pods like these should run: There some preceding discussion in the "sig-node" channel of the "Kubernetes" Slack team . Possibly related issues: , Possibly related PRs: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS EC2 (g3.8xlarge instance, but other instance types exhibit the same behavior) OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: kubeadm init, kubeadm join
/sig node /sig apps
To work around this problem, I found that I can use multiple match expressions, with one candidate value in each. This works because sibling match expressions are a disjunction.
if the kubelet is rejecting the pod, then this belongs to sig-node it might be reusing logic from the scheduler library, so looping in sig-scheduling for reference: /sig scheduling
I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. /assign
GitHub didn't allow me to assign the following users: Huang-Wei. Note that only and repo collaborators can be assigned. <detailsIn response to : I can reproduce it on kubeadm v1.11.0. Let me try to give a fix. </details
I've found the root cause, and should be able to send out a PR next week.


Question: _ "k8s.io/kubernetes/pkg/api/v1" _ "k8s.io/kubernetes/pkg/apis/extensions" _ "k8s.io/kubernetes/pkg/apis/extensions/v1beta1" <del> pkg_runtime "k8s.io/kubernetes/pkg/runtime" </del> <ins> kruntime "k8s.io/kubernetes/pkg/runtime" </ins> "k8s.io/kubernetes/pkg/util/sets" "github.com/golang/glog"

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: } } } <ins> func TestIsCriticalPodBasedOnPriority(t *testing.T) { tests := []struct { priority    int32 description string expected    bool }{ { priority:    int32(2000000001), description: "A system critical pod", expected:    true, }, { priority:    int32(1000000000), description: "A non system critical pod", expected:    false, }, } for _, test := range tests { actual := IsCriticalPodBasedOnPriority(test.priority) if actual != test.expected { t.Errorf("IsCriticalPodBased on priority should have returned %v for test %v but got %v", test.expected, test.description, actual) } } } </ins>

When we used annotations to mark critical pods, we considered pods with such an annotation critical only when they were created in "kube-system" namespace. Going to the new era of using priority to indicate critical pods, we would like to remove the restriction of "kube-system" namespace and consider pods with critical priority as critical no matter what namespace they are in. ref/ /kind bug /sig scheduling /assign cc/
While I am fine with the approach, it seems the only way to ensure that those critical pods are not coming from some random namespace is whitelisting them using priority admission controller or using quota. Atleast previously we have one more level of security that critical pod has to be created in kube-system namespace.
The requirement to limit the critical pod annotation meaning to the kube-system namespace was a workaround for the lack of access control and quota around the annotation. We actively work to prevent special handling of the kube-system namespace wherever possible.
Yes, which is why those aspects are essential to a functional priority system that works for a variety of cluster configurations
Agree with Ravi and Jordan on what is needed for final solution.
There is no doubt that we need to have quota for priority. It is needed not only for critical priorities, but also for any other high priority classes that users may define. That effort is going forward and we will have it soon.
, I think we also need to update Priority admission controller for that.
see later PR re-restricting this until the quota controls graduate to at least beta state:
Got that, thanks :) That makes sense to me.
If this is closed and there even was a for this type of functionality, what is the current status? And is in sync with current status?
It seems that namespace is still , so I guess the current status is that system critical pods are still limited to namespace. Based on that I think this issue should not be closed.
can we reopen this issue please because as pointed out by the issue still exists. Thanks.
Waiting for this to be available myself. No reason my Node Exporter dataset has to run in the namespace instead of my Grafana namespace.
Confirmed code + behavior. /open
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I too am confused based on that documentation. Can we get an update here with an answer and fix the documentation if it's wrong?
This is resolved in 1.17 See


Question: RUN find /go/bin -name sample-apiserver -exec cp {} / ; FROM BASEIMAGE <del> COPY --from=build_k8s_1_7_sample_apiserver /sample-apiserver /sample-apiserver </del> <ins> COPY --from=build_k8s_1_10_sample_apiserver /sample-apiserver /sample-apiserver </ins> ENTRYPOINT ["/sample-apiserver"]

[sig-api-machinery] Aggregator Should be able to support the 1.7 Sample API Server using the current Aggregator More nuanced logs and details can be found at gave up waiting for apiservice wardle to come up successfully Expected error: <errors.errorString | 0xc420131150: { s: "timed out waiting for the condition", } timed out waiting for the condition not to have occurred /priority failing-test /priority important-soon /kind flake /sig-api-machinery /assign can you please help triage this issue?
has made changes that could have affected this recently. made one that'd be even more suspicious but I don't think it has merged yet :) :)
flakes show steady 2-3/day over the past week: which changes are you referring to? the line that's failing isn't doing any discovery, restmapping, etc, at all... just a straight get to the API:
audit log from the test run shows the requests are returning 404s, not 503s
ah, sorry, I misread the log and thought it was a discovery error. Hm.
hmm, there's a second smaller set of failures that are discovery related: looks like those started flaking after which went in on 5/4 looks like some discovery paths had internal retries that could have masked those 404 errors before. since the error that test is encountering is for an unrelated CRD API that is being created/deleted as part of another test, it makes more sense to switch that check to just ensure the extension API group under test is discovered, rather than assert there are no errors looking for other API groups that we know are being dynamically /removed opened to deflake the aggregator e2e check
/assign resolved the discovery issue. now the failure is the one noted in where calls to the extension API are responding with 404 errors
Kubelet log is reporting that the sample-apiserver exited with a return code of 2 and it also shows with a high restart count for the sample-apiserver.

When I extract the sample-apiserver log I'm seeing multiple "panic: Get https://10.0.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication: dial tcp 10.0.0.1:443: getsockopt: connection refused" Both failures today are in ipvs clusters which seems to have extensive networking problems. /assign
We are looking into finding the root cause into the problems with the ipvs clusters. I am pretty confident they were triggered by a bad PR, just need to identify which one.
Initial hypothesis is that PR caused it.
We think should fix the IPVS issues.
I see merged and this test is now passing in both pull-kubernetes-e2e-gce and ci-kubernetes-e2e-gci-gce. Hence closing the issue. Thanks all
/reopen The test is failing for the same error again in the following jobs https://k8s- It is one of the top flake from last week failing 47 jobs.
can you please take a look and triage? Thanks
this flake is now blocking this test from being promoted to Conformance suite in 1.11. Is it something we can investigate and resolve soon (today or tomorrow) to increase changes of getting this into 1.11? If not this test has to wait until 1.12 to be promoted.
all I see in the test run logs is 404 errors coming back from that check. can you see anything useful in the container logs for the aggregated server?
/cc
Update: I have been trying to reproduce this failure on my own test cluster (go run ...) and it has passed 128 times in a row so far. I think this means that the issue must be with either an interaction between e2e tests or an issue with the job running the tests.
Looking through recent results this crops up as a flake in presubmit and CI across many configurations... I don't think how we run the test jobs should be affecting this in the slightest. Not sure what's wrong here, but it looks like a pretty typical flakey test (some race / buggy setup/teardown?) nb that may not trivially map to the same way the tests are run though, and itself is a wrapper over
does the API server logs throw any light on whats happening on CI test cluster.

The teardown for the aggregator test is wrong, it tries to delete the deployment "sample-apiserver" when the deployment is actually named "sample-apiserver-deployment", but I don't think that's the issue. It's been like that since the test was . I'll look for conflicts with other tests.
We should probably go ahead and fix that anyway, it might be stomping on another test... Jenny Buckley <:
The namespace controller polls each of the api groups periodically, and it gets this internal error back from the endpoint a couple of times. Maybe related?
unclear... I opened to capture the state of the APIService, extension server pods, and those pod logs in failures cases
It flaked again since your PR merged https://k8s-
All I'm seeing is
I don't think that's related... I see 10-30 of those same rejections in apiserver logs of successful test runs (for example, search for ). I don't see anything in the test setup that grants the sample apiserver permission to run those checks, so it doesn't seem like that would be intermittent or flaky.
I am re-opening this issue since I still see intermittent failure of this test are your pending PRs ( and ) targeted towards these remaining issues? /reopen
this looks like 1.7 API Server image is causing more flakes other than just this test - The test has become lot less flaky after increasing the timeout. Since the official policy for GKE is 2 previous versions, and 1.11 is almost baked, this is the plan discussed with offline: leave the 1.7 test as it is right now create a similar test to the 1.7, but with 1.9 (implies creating a new sample-sever, uploading image, and creating the test). get that new test PR merged and running let it soak for several days, see if in 1.7 - 1.9 the flakiness (latency or whatever is causing it) went away.

if it did, evaluate removing the 1.7 and using the 1.9 let us know what you think of this approach
I think you misunderstood my comment... that only related to kubectl being used against a 1.7-level kube-apiserver serving discovery docs for the v1 API, not a generic 1.7-level extension API server
unrelatedly, moving the test to 1.9-level code once 1.11 is released seems reasonable to me
Thank you Jordan, that's the current plan! We will actually create a parallel 1.9-level test, to see if between 1.7 and 1.9 the internals of API Server fixed what's generating the flakiness and see next steps from there.
Sorry I didn't read the previous discussion close enough. The idea of creating a 1.9 test makes sense.
Pinging to see what are the next steps. Who is taking point to create a parallel test?
last I chated with someone on his team will build 1.9 API server image and push it to GCR. Once that image is available then can help writing the parallel test. I am not sure if the image step is done.
It's on the works. Buiding the sample-image showed to be more complicated than what it seemed to be. If it becomes a blocker let me know. Aish Sundar <:
I'd like to know where we are on this, because the 1.7 thing has come up in the context of multi-arch image builds
and
Hi Aaron and Aish, sorry for the delay. The process to compile, create and upload the sample API Server was not properly documented / maintained, and that plus more burning things is what made it take so long. We are trying to finally have it back (and documented by end of this week). Will keep you posted.
Image created with all the refactor happened on that area, trying upgrading the test at present from 1.7 to 1.9
Update: Upgraded the test, and now the test is not passing... working on this.

• Failure [38.067 seconds] [sig-api-machinery] Aggregator Should be able to support the 1.9 Sample API Server using the current Aggregator [It] creating a new flunders resource Expected error: <errors.StatusError | 0xc420c90cf0: { ErrStatus: { TypeMeta: {Kind: "", APIVersion: ""}, ListMeta: {SelfLink: "", ResourceVersion: "", Continue: ""}, Status: "Failure", Message: " "rest-flunder-" is forbidden: not yet ready to handle request", Reason: "Forbidden", Details: { Name: "rest-flunder-", Group: "", Kind: "flunders", UID: "", Causes: nil, RetryAfterSeconds: 0, }, Code: 403, }, } "rest-flunder-" is forbidden: not yet ready to handle request not to have occurred
not sure if it's related, but I don't think the test currently gives sufficient permissions to the aggregated server to perform delegated authn/authz checks. we might want to look at merging
tried your fix, same result. The image of the sample-apiserver 1.9 based that I've created is uploaded to If someone want's to give it a try, simply by upgrading 1.0 to 1.1 here:
/assign Can you try to figure out why the test is failing w/ the new image?
/cc
Now waiting on
Is there an ETA on ? This test is flaking quite a bit offlate - https://k8s-
upgrades to 1.10. however it requires more rbac permissions than before which is not fully understood. We will either resolve the permission issue or leave it for later investigation. The upgrade to 1.10 should be merged in a couple of days, if not today.
what issue are you seeing? is relevant?
Yes, it is relevant. It does not fix the issue by itself though.
As for should be ready to be merged.
/reopen I would like some manual confirmation this has been addressed
Reopening this issue. <detailsIn response to : /reopen I would like some manual confirmation this has been addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
after some in-depth investigations, it turns out the 1.10 sample-apiserver would need the following rbac permissions (in addition to the system:auth-delegator role). I'm updating the PR to add these. apiVersion: kind: ClusterRole metadata: name: sample-apiserver rules: apiGroups: resources: '*' verbs: list apiGroups: '' resources: namespaces verbs: list
This test is failing again, FYI: https://k8s-
Hi Josh, TL;DR: we are upgrading to 1.10 and that's what people is talking about in the previous comments.
/reopen I would like to see the test pass with the fix before closing this issue out. thanks
Reopening this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi please notice the test was renamed to: [sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator, and has been passing since merged? https://k8s-
Ah cool thanks that explains it :) I was looking at in release-master-blocking and looks like it hasn't picked by the latest test since its 10/13 run. Closing this issue now based on the api-machinery dashboard now. Will reopen if the release job doesnt turn green the next run. Thanks much.
Is it worth removing the copy of this test that lives in ? Or is it still a release-blocking test?
I think the test will automatically pick the new version of the test in its next run (which should be sometime today, 10/20?). As to whether the test should be in the scale-correctness job , it is a ques for and


Question: epCount := len(config.EndpointPods) config.MaxTries = epCount*epCount + testTries <ins> framework.Logf("Setting MaxTries for pod polling to %v for networking test based on endpoint count %v", config.MaxTries, epCount) </ins> } // setup includes setupCore and also sets up services

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: } // getContainersToDeleteInPod returns the exited containers in a pod whose name matches the name inferred from exitedContainerID, ordered by the creation time from the latest to the earliest. <del> func (p *podContainerDeletor) getContainersToDeleteInPod(exitedContainerID string, podStatus *kubecontainer.PodStatus) containerStatusbyCreatedList { </del> <ins> func getContainersToDeleteInPod(exitedContainerID string, podStatus *kubecontainer.PodStatus, containersToKeep int) containerStatusbyCreatedList { </ins> var matchedContainer *kubecontainer.ContainerStatus var exitedContainers []*kubecontainer.ContainerStatus // Find all exited containers in the pod

Kubelet evicts pods whenever there is memory or disk pressure. It marks the pods as to evict them. To help with reducing disk pressure, kubelet should delete all containers belonging to pods right away. This will result in loss of logs. In the future, we can copy out logs from dead containers before deleting them. Logs can be rotated separately from dead containers. IMO, this feature is required for disk usage based evictions to work reliably in kubelet.


Question: // Slow by design. var _ = SIGDescribe("Node Performance Testing [Serial] [Slow]", func() { f := framework.NewDefaultFramework("node-performance-testing") <ins> var ( wl     workloads.NodePerfWorkload oldCfg *kubeletconfig.KubeletConfiguration newCfg *kubeletconfig.KubeletConfiguration pod    *corev1.Pod ) JustBeforeEach(func() { err := wl.PreTestExec() framework.ExpectNoError(err) oldCfg, err = getCurrentKubeletConfig() framework.ExpectNoError(err) newCfg, err = wl.KubeletConfig(oldCfg) framework.ExpectNoError(err) setKubeletConfig(f, newCfg) }) </ins> <del> Context("Run node performance testing with pre-defined workloads", func() { It("run each pre-defined workload", func() { By("running the workloads") for _, workload := range workloads.NodePerfWorkloads { By("running the pre test exec from the workload") err := workload.PreTestExec() framework.ExpectNoError(err) By("restarting kubelet with required configuration") // Get the Kubelet config required for this workload. oldCfg, err := getCurrentKubeletConfig() framework.ExpectNoError(err) newCfg, err := workload.KubeletConfig(oldCfg) framework.ExpectNoError(err) // Set the Kubelet config required for this workload. setKubeletConfig(f, newCfg) By("running the workload and waiting for success") // Make the pod for the workload. pod := makeNodePerfPod(workload) // Create the pod. pod = f.PodClient().CreateSync(pod) // Wait for pod success. f.PodClient().WaitForSuccess(pod.Name, workload.Timeout()) podLogs, err := framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, pod.Spec.Containers[0].Name) framework.ExpectNoError(err) perf, err := workload.ExtractPerformanceFromLogs(podLogs) framework.ExpectNoError(err) framework.Logf("Time to complete workload %s: %v", workload.Name(), perf) // Delete the pod. gp := int64(0) delOpts := metav1.DeleteOptions{ GracePeriodSeconds: &gp, } f.PodClient().DeleteSync(pod.Name, &delOpts, framework.DefaultPodDeletionTimeout) </del> <ins> cleanup := func() { gp := int64(0) delOpts := metav1.DeleteOptions{ GracePeriodSeconds: &gp, } f.PodClient().DeleteSync(pod.Name, &delOpts, framework.DefaultPodDeletionTimeout) By("running the post test exec from the workload") err := wl.PostTestExec() framework.ExpectNoError(err) setKubeletConfig(f, oldCfg) } </ins> <del> By("running the post test exec from the workload") err = workload.PostTestExec() framework.ExpectNoError(err) </del> <ins> runWorkload := func() { By("running the workload and waiting for success") // Make the pod for the workload. pod = makeNodePerfPod(wl) // Create the pod. pod = f.PodClient().CreateSync(pod) // Wait for pod success. f.PodClient().WaitForSuccess(pod.Name, wl.Timeout()) podLogs, err := framework.GetPodLogs(f.ClientSet, f.Namespace.Name, pod.Name, pod.Spec.Containers[0].Name) framework.ExpectNoError(err) perf, err := wl.ExtractPerformanceFromLogs(podLogs) framework.ExpectNoError(err) framework.Logf("Time to complete workload %s: %v", wl.Name(), perf) } </ins> <del> // Set the Kubelet config back to the old one. setKubeletConfig(f, oldCfg) } </del> <ins> Context("Run node performance testing with pre-defined workloads", func() { BeforeEach(func() { wl = workloads.NodePerfWorkloads[0] }) It("NAS parallel benchmark (NPB) suite - Integer Sort (IS) workload", func() { defer cleanup() runWorkload() }) }) Context("Run node performance testing with pre-defined workloads", func() { BeforeEach(func() { wl = workloads.NodePerfWorkloads[1] }) It("NAS parallel benchmark (NPB) suite - Embarrassingly Parallel (EP) workload", func() { defer cleanup() runWorkload() }) }) Context("Run node performance testing with pre-defined workloads", func() { BeforeEach(func() { wl = workloads.NodePerfWorkloads[2] }) It("TensorFlow workload", func() { defer cleanup() runWorkload() </ins> }) }) })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: test/e2enode/perf tests are referring to adm only images, this need to be replaced with multi-arch fat manifest images. ref:
/assign /sig node


Question: } } <ins> // ensureBridgeTxQueueLen() ensures that the bridge interface's TX queue // length is greater than zero.  Due to a CNI <= 0.3.0 'bridge' plugin bug, // the bridge is initially created with a TX queue length of 0, which gets // used as the packet limit for FIFO traffic shapers, which drops packets. // TODO: remove when we can depend on a fixed CNI func (plugin *kubenetNetworkPlugin) ensureBridgeTxQueueLen() { bridge, err := netlink.LinkByName(BridgeName) if err != nil { return } if bridge.Attrs().TxQLen > 0 { return } req := nl.NewNetlinkRequest(syscall.RTM_NEWLINK, syscall.NLM_F_ACK) msg := nl.NewIfInfomsg(syscall.AF_UNSPEC) req.AddData(msg) nameData := nl.NewRtAttr(syscall.IFLA_IFNAME, nl.ZeroTerminated(BridgeName)) req.AddData(nameData) qlen := nl.NewRtAttr(syscall.IFLA_TXQLEN, nl.Uint32Attr(1000)) req.AddData(qlen) _, err = req.Execute(syscall.NETLINK_ROUTE, 0) if err != nil { glog.V(5).Infof("Failed to set bridge tx queue length: %v", err) } } </ins> func (plugin *kubenetNetworkPlugin) Name() string { return KubenetPluginName }

I'm not sure where the defaults come from but they sometimes drop all packets causing container IPs to become unreachable. ARP fails. I'm not sure if this is related but I also noticed on this machine that the default pfifo qdisc drops all packets. pfifo is the default qdisc for an htb node so it's used in the unconfigured htb qdisc: This is because the default value for queue size is zero. On other machines I've seen the default value set to 1 packet and 1000 packets. When I set the limit explicitly the container becomes reachable again. The broken configuration is applied by kubelet to implement traffic shapping and there is no option to turn this off. This patch fixes the kubelet on broken machines by disabling traffic shapping: cc
does this happen with --configure-cbr0 and packet shaping too? The kubenet implementation just uses the existing kubernetes shaping code that --configure-cbr0 does.
It seems the *fifo qdiscs default their packet limit to the txquelen of the interface (man tc-pfifo). It also seems most virtual interface types (bridge, etc) set a dev-txqueuelen of 0. It further seems that relies on the default packet limit. I could swear this worked before with kubenet, and it must have worked with --configure-cbr0 when the code was originally . I'll check around to figure out what the default limit should be for interfaces that don't specify a tx queue length.
Note that to test with --configure-cbr0 you'd also need --reconcile-cidr.
Upstream CNI fix is here: and we'll also work around it in kubenet for now.
FWIW, the upstream CNI fix got merged for CNI 0.3.0


Question: SystemUUID string `json:"systemUUID"` // BootID is the boot-id reported by the node BootID string `json:"bootID" description:"boot id is the boot-id reported by the node"` <ins> // Kernel version reported by the node KernelVersion string `json:"kernelVersion" description:"Kernel version reported by the node from 'uname -r' (e.g. 3.16.0-0.bpo.4-amd64)"` // OS image used reported by the node OsImage string `json:"osImage" description:"OS image used reported by the node from /etc/os-release (e.g. Debian GNU/Linux 7 (wheezy))"` // Container runtime version reported by the node ContainerRuntimeVersion string `json:"containerRuntimeVersion" description:"Container runtime version reported by the node through runtime remote API (e.g. docker://1.5.0)"` // Kubelet version reported by the node KubeletVersion string `json:"kubeletVersion" description:"Kubelet version reported by the node"` // Kube-proxy version reported by the node KubeProxyVersion string `json:"KubeProxyVersion" description:"Kube-proxy version reported by the node"` </ins> } // NodeStatus is information about the current status of a node.

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: According to the two changes are required to enable it: Add to apiserver and scheduler. Add to apiserver. So I modified the apiserver manifest: And the scheduler manifest: confirms that both are running with the required changes. However, I cannot create PodPriorities by following the example in the documentation: also doesn't think the API is enabled: The only relevant message in the apiserver log is: What you expected to happen: I expected to be able to create podpriority resources by following the example in the documentation. How to reproduce it (as minimally and precisely as possible): See description. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): CentOS Linux 7.4 (1708) Kernel (e.g. ): Linux master-head-1 4.13.4- SMP Wed Sep 27 13:32:23 EDT 2017 x8664 x8664 x8664 GNU/Linux Install tools: Custom scripts Others:
/sig scheduling
The example yaml of priorityclass is wrong, the right version is
please try the recommendation from and close this issue?
unfortunately that version change to the YAML does not fix the problem for me either: To me it looks like the error message points me to where I don't see defined?
Yes, you are right, I have commit a pr to fix this.
Not sure but after using the updated config I am able to create Priority Class successfully but unable to use the feature while creating POD. Here are the errors observed Getting following error while creating pod: The PodPriority is enabled in both API server and Scheduler
I'm able to create PriorityClass and schedule pod with specified PriorityClass on Minikube 0.26.0. ` !
This issue was sorted - on kubeadm production setup, this flag needs to be enabled for kubelet as well - updated doc
I've created the PriorityClass and deployed the pod with the specified PriorityClass. The pod is getting created. I'm wondering is there any way to check the PriorityClass of the pod either in UI or CLI? The steps you've shown is giving me the same result. Can you please help me ? Thanks in advance.


Question: "k8s.io/client-go/util/workqueue" "k8s.io/kubernetes/pkg/printers" <ins> utilversion "k8s.io/kubernetes/pkg/util/version" </ins> "k8s.io/kubernetes/test/e2e/framework" imageutils "k8s.io/kubernetes/test/utils/image" ) <ins> var serverPrintVersion = utilversion.MustParseSemantic("v1.10.0") </ins> var _ = SIGDescribe("Servers with support for Table transformation", func() { f := framework.NewDefaultFramework("tables") <ins> BeforeEach(func() { framework.SkipUnlessServerVersionGTE(serverPrintVersion, f.ClientSet.Discovery()) }) </ins> It("should return pod details", func() { ns := f.Namespace.Name c := f.ClientSet

the tests are failing in http://k8s- /sig sig-api-machinery /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone /milestone v1.10 cc /assign
The provided milestone is not valid for this repository. Here are the available milestones: 1.clear 1.v1.6 1.v1.13 1.v1.10 1.v1.11 1.v1.8 1.v1.4 1.v1.5 1.v1.12 1.next-candidate 1.v1.9 1.v1.7 <detailsIn response to : the tests are failing in http://k8s- /priority failing-test /priority critical-urgent /kind bug /status approved-for-milestone /milestone v1.10 /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
xref
/sig api-machinery
/assign /assign
The three tests in should be run only on 1.10+ servers. I'm just putting a PR together, should be up within next hour or so.
Master fix is in .
/reopen still failing in upgrade tests: http://k8s- the 1.9 e2e tests are checking alpha response types that don't exist any more in 1.10 requires changing 1.9 to not test these alpha features in 1.10 that dropped support for those alpha features
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen still failing in upgrade tests: http://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is there anyone from that can look into this?
looking into this
/status in-progress
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed.

: Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
fixed by


Question: controlPlaneConfig := framework.NewIntegrationTestControlPlaneConfig() controlPlaneConfig.GenericConfig.Authentication.Authenticator = authenticator.RequestFunc(func(req *http.Request) (*authenticator.Response, bool, error) { return &authenticator.Response{ <del> User: &user.DefaultInfo{Name: username}, </del> <ins> User: &user.DefaultInfo{ Name:   username, UID:    username, Groups: []string{user.AllAuthenticated}, }, </ins> }, true, nil }) controlPlaneConfig.GenericConfig.Authorization.Authorizer = sarAuthorizer{}

While moving APF to beta (see comments on ), we discovered that the integration tests commonly present users which are not members of any groups. We had thought that the system enforced membership in either the authenticated or unauthenticated group. If this is supposed to be a system invariant (as we thought), we'd like to ask SIG Auth: Please fix the integration tests Please add e.g. logging so that if real, operational clusters hit this there's at least a log message and people will file bugs. /sig auth
Can I suggest also, if this is a real invariant, that it be documented in some place(s) likely to be read?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale /priority important-longterm /triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/lifecycle frozen
fixes the authenticator configured in one of the integration test API server startup helpers which resolved this issue for . Remaining tests that still need fixing that are encountering unexpected users look to be ones set up to specifically test auth paths:
open to fix remaining integration test users


Question: } } <ins> func TestDeleteWithGCFinalizers(t *testing.T) { storage, server := newStorage(t) defer server.Terminate(t) defer storage.store.DestroyFunc() propagationBackground := metav1.DeletePropagationBackground propagationForeground := metav1.DeletePropagationForeground propagationOrphan := metav1.DeletePropagationOrphan trueVar := true var tests = []struct { name          string deleteOptions *metav1.DeleteOptions existingFinalizers  []string remainingFinalizers map[string]bool }{ { name:          "nil-with-orphan", deleteOptions: nil, existingFinalizers: []string{ metav1.FinalizerOrphanDependents, }, remainingFinalizers: map[string]bool{ metav1.FinalizerOrphanDependents: true, }, }, { name:          "nil-with-delete", deleteOptions: nil, existingFinalizers: []string{ metav1.FinalizerDeleteDependents, }, remainingFinalizers: map[string]bool{ metav1.FinalizerDeleteDependents: true, }, }, { name:                "nil-without-finalizers", deleteOptions:       nil, existingFinalizers:  []string{}, remainingFinalizers: map[string]bool{}, }, { name: "propagation-background-with-orphan", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationBackground, }, existingFinalizers: []string{ metav1.FinalizerOrphanDependents, }, remainingFinalizers: map[string]bool{}, }, { name: "propagation-background-with-delete", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationBackground, }, existingFinalizers: []string{ metav1.FinalizerDeleteDependents, }, remainingFinalizers: map[string]bool{}, }, { name: "propagation-background-without-finalizers", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationBackground, }, existingFinalizers:  []string{}, remainingFinalizers: map[string]bool{}, }, { name: "propagation-foreground-with-orphan", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationForeground, }, existingFinalizers: []string{ metav1.FinalizerOrphanDependents, }, remainingFinalizers: map[string]bool{ metav1.FinalizerDeleteDependents: true, }, }, { name: "propagation-foreground-with-delete", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationForeground, }, existingFinalizers: []string{ metav1.FinalizerDeleteDependents, }, remainingFinalizers: map[string]bool{ metav1.FinalizerDeleteDependents: true, }, }, { name: "propagation-foreground-without-finalizers", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationForeground, }, existingFinalizers: []string{}, remainingFinalizers: map[string]bool{ metav1.FinalizerDeleteDependents: true, }, }, { name: "propagation-orphan-with-orphan", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationOrphan, }, existingFinalizers: []string{ metav1.FinalizerOrphanDependents, }, remainingFinalizers: map[string]bool{ metav1.FinalizerOrphanDependents: true, }, }, { name: "propagation-orphan-with-delete", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationOrphan, }, existingFinalizers: []string{ metav1.FinalizerDeleteDependents, }, remainingFinalizers: map[string]bool{ metav1.FinalizerOrphanDependents: true, }, }, { name: "propagation-orphan-without-finalizers", deleteOptions: &metav1.DeleteOptions{ PropagationPolicy: &propagationOrphan, }, existingFinalizers: []string{}, remainingFinalizers: map[string]bool{ metav1.FinalizerOrphanDependents: true, }, }, { name: "orphan-dependents-with-orphan", deleteOptions: &metav1.DeleteOptions{ OrphanDependents: &trueVar, }, existingFinalizers: []string{ metav1.FinalizerOrphanDependents, }, remainingFinalizers: map[string]bool{ metav1.FinalizerOrphanDependents: true, }, }, { name: "orphan-dependents-with-delete", deleteOptions: &metav1.DeleteOptions{ OrphanDependents: &trueVar, }, existingFinalizers: []string{ metav1.FinalizerDeleteDependents, }, remainingFinalizers: map[string]bool{ metav1.FinalizerOrphanDependents: true, }, }, { name: "orphan-dependents-without-finalizers", deleteOptions: &metav1.DeleteOptions{ OrphanDependents: &trueVar, }, existingFinalizers: []string{}, remainingFinalizers: map[string]bool{ metav1.FinalizerOrphanDependents: true, }, }, } for _, test := range tests { key := "namespaces/" + test.name ctx := genericapirequest.NewContext() namespace := &api.Namespace{ ObjectMeta: metav1.ObjectMeta{ Name:       test.name, Finalizers: test.existingFinalizers, }, Spec: api.NamespaceSpec{ Finalizers: []api.FinalizerName{}, }, Status: api.NamespaceStatus{Phase: api.NamespaceActive}, } if err := storage.store.Storage.Create(ctx, key, namespace, nil, 0, false); err != nil { t.Fatalf("unexpected error: %v", err) } var obj runtime.Object var err error if obj, _, err = storage.Delete(ctx, test.name, test.deleteOptions); err != nil { t.Fatalf("unexpected error: %v", err) } ns, ok := obj.(*api.Namespace) if !ok { t.Errorf("unexpected object kind: %+v", obj) } if len(ns.Finalizers) != len(test.remainingFinalizers) { t.Errorf("%s: unexpected remaining finalizers: %v", test.name, ns.Finalizers) } for _, f := range ns.Finalizers { if test.remainingFinalizers[f] != true { t.Errorf("%s: unexpected finalizer %s", test.name, f) } } } } </ins> func TestShortNames(t *testing.T) { storage, server := newStorage(t) defer server.Terminate(t)

What happened: I created a namespace with finalizer, and then tried to delete it: The namespace can be deleted, and the finalizer is removed by garbage collection controller: What you expected to happen: Since kubectl species , it is not an orphaning operation, the finalizer should be removed in namespace store. How to reproduce it (as minimally and precisely as possible): Same as what I observed: a namespace. finalizer. . Anything else we need to know?: Environment: Kubernetes version (use ): HEAD
/sig api-machinery
cc
Discussion on slack:


Question: <ins> /* Copyright 2022 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ // This file helps to detect whether we changed the interface of Framework. // This is important for projects under `github.com/kubernetes*` orgs // who import the Framework to not breaking their codes, e.g. cluster-autoscaler. package contract import ( "context" "testing" "github.com/stretchr/testify/assert" v1 "k8s.io/api/core/v1" "k8s.io/kubernetes/pkg/scheduler/framework" "k8s.io/kubernetes/pkg/scheduler/framework/runtime" ) type frameworkContract interface { RunPreFilterPlugins(ctx context.Context, state *framework.CycleState, pod *v1.Pod) (*framework.PreFilterResult, *framework.Status) RunFilterPlugins(context.Context, *framework.CycleState, *v1.Pod, *framework.NodeInfo) framework.PluginToStatus } func TestFrameworkContract(t *testing.T) { var f framework.Framework var c frameworkContract = f assert.Nil(t, c) } func TestNewFramework(t *testing.T) { var f interface{} if f, _ = runtime.NewFramework(nil, nil, nil); f != nil { _, ok := f.(framework.Framework) assert.True(t, ok) } } func TestNewCycleState(t *testing.T) { var state interface{} = framework.NewCycleState() _, ok := state.(*framework.CycleState) assert.True(t, ok) } </ins>

Unit test that will ensure that Cluster Autoscaler objects that currently implement interfaces from stay that way. It would suffice to do add compile-time checks for the following types: Interface changes could still be introduced in a PR against k/k repo, but they would require Cluster Autoscaler changes first to be merged before the interface change itself is merged. To avoid circular dependency issues, the unit test itself would have to live in a separate module inside k/k repo, importing both scheduler interfaces and CA implementations. Scheduler framework is an API used for communication between Cluster Autoscaler and Scheduler code. Since it is not a versioned API, it is easy to introduce changes that are backwards incompatible, which is exactly what happened recently in Ensuring interface compatibility in core k8s repo will prevent accidentally breaking Cluster Autoscaler code.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
/sig autoscaling
I'm +1 and based on the discussion on last sig-autoscaling meeting the idea has a wide support in the sig-autoscaling. One question: shouldn't we also cover ?
Good point, CA is not only implementing scheduler interfaces, but also calling them, I think it makes sense to extend the scope to include in a compile-time check.
+1 I think it's not only for CA. But also for the plugins (not in k/k) implemented by users themselves. Ensuring interface compatibility is needed.
I understand the intention to prevent CA's code to be broken, however, the "CA's change first" strategy doesn't sound that practical as (1) that would require k/k to vendor CA's code, and (2) the "preventive changes" in CA doesn't really get verified at runtime/compilation time as the upstream changes are not merged yet. Given no matter what kind of changes go first, the scheduler framework changes need to get CA team involved. I'd suggest we continue to follow this rule, which works well all the time and is just violated once in .

In addition to the early involvement in the k/k PR's review, we can setup additional CI check in CA's side, so when the deps in CA, the CI should have (which isn't today).
The scheduler framework changes are somewhat unavoidable to evolve over time, aimed to behave performant as well as fitting more scenarios. The changes vary from minor changes like PodInfo-QueuedPodInfo, to major changes like interface addition/deletion/update, and hence those would require the consumers (like plugins) to adjust when bumping versions. I don't think we can promise strict compatibility at this moment.
Good point that we wouldn't be able to verify the code on CA side does indeed implement the updated interface. That might be sufficient, I'm just worried that this is not an automated process and hence can be skipped again. Is there anything we could do to limit the possibility of human error? My best idea here so far is to add sig-autoscaling as reviewers for framework changes. Or perhaps a subset of sig-autoscaling members that are cluster-autoscaler maintainers (as opposed to, for example, VPA). Actually, tests are already run for CA PRs, so it should already be detectable that way. The problem is that by the point it happens, the interface changes are already merged and there is no way to influence them to be more compatible with CA use case.
I think this is where unittests could help. The unittest wouldn't mean the interface cannot be changed, it would just provide a signal that sig-autoscaling / Cluster Autoscaler reviewer should be involved. It seems much more reliable than the current system based on scheduler contributors' intuitions on which changes may affect CA.
+1 on unit tests owned by sig-autoscaling
Unit test will block submission, which is more than just letting sig-autoscaling know they should take a look. In that case an integration test would probably make more sense: use GitHub API to pull latest version of kubernetes/autoscaler repo, then use Go compiler libs to try to compile selected files. On breaking interface changes, SIG autoscaling approver would be required to temporarily disable the test and drive adjustments on Cluster Autoscaler side (asynchronously, not blocking development on k/k repo). Thoughts?
I think we just need to have tests in k/k that hardcode the CA expectations rather than trying to think how to import CA.

Do you mean a slim fake library to simulate CA and other consumers? If so, it may still have gaps with CA. While if we import CA in k/k, again, that may yield a cyclic dependency. That sounds fair to me as it simulates how a consumer consumes the framework. Just not sure how complicated it can be (supposing we don't have similar tests to leverage).
I am hoping that we don't need to "simulate" CA; just their expectations of the framework itself.
I like suggestion: Simple test that validates CA expectations of the framework interface, without taking on all the extra complexity stemming from dependencies. This should be sufficient for now and may be replaced by more sophisticated mechanism I described above if needed. At the very least, it should suffice to create an object and check if it satisfies the framework interface. We can iterate on that later, if needed.
+1 with the suggestion. What about verify the framework at startup like , and make CA team part of the folder reviewer.
Yup, exactly something like this. I think putting it inside a unit test makes more sense though - there's no reason for this code to end up in scheduler binary.
would you like to finish that PR?
Sure, quite busy in the past few days. I'll complete this ASAP. Thanks for reminding.
Thanks, I left a minor comment on the PR, otherwise LGTM.


Question: } return matches[1], nil } <ins> // Replace topology values for failure domains ("<number>") to "", // as it's the value that the CSI driver expects. func (t *azureDiskCSITranslator) replaceFailureDomainsToCSI(terms []v1.TopologySelectorTerm) []v1.TopologySelectorTerm { if terms == nil { return nil } newTopologies := []v1.TopologySelectorTerm{} for _, term := range terms { newTerm := term.DeepCopy() for i := range newTerm.MatchLabelExpressions { exp := &newTerm.MatchLabelExpressions[i] if exp.Key == AzureDiskTopologyKey { for j := range exp.Values { if unzonedCSIRegionRE.Match([]byte(exp.Values[j])) { // Topologies "0", "1" etc are used when in-tree topology is translated to CSI in Azure // regions that don't have availability zones. E.g.: //    topology.kubernetes.io/region: westus //    topology.kubernetes.io/zone: "0" // The CSI driver uses zone "" instead of "0" in this case. //    topology.disk.csi.azure.com/zone": "" exp.Values[j] = "" } } } } newTopologies = append(newTopologies, *newTerm) } return newTopologies } </ins>

This test fails on Kubernetes cluster in "westus" region (without any number) and with CSI migration enabled: Error: failed to provision volume with StorageClass "e2e-topology-84vk7sp": error generating accessibility requirements: topology map[] from selected node "ci-op-9j1nby77--9zlwv-worker-westus-ljw4g" is not in requisite: [map[]] StorageClass generated by the test: PVC created by the test: <details</detailsLabels of the selected node: The external-provisioner does not even call the CSI driver and reports this event: The volume should get provisioned. See above, "westus" region with no availability zones seems to be the key. It looks like the translation from in-tree StorageClass to CSI one translates topology requirements to , but the CSI one expects <details</details<details</details<detailsOpenShift 4.10 </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority important-soon cc
/sig storage
per https://kubernetes-, is not a zone, it's actually a fault-domain num. valid zone format is , could you change the storage class with setting, and try again? Thanks.
o, if "westus" region does not support zone, then I think you should test setting in another zone, make sure you have created a zoned node first.
I have as SC that works without CSI migration, with CSI migration it does not work. That's a regression. It should not matter if the SC is somehow odd. The labels are selected by test code, some other production code might use these labels too.
Out of curiosity, how reliable are zone numbers in Azure? I can see .. in a region with zones. If is used only in regions without a zone, then the translation can translate to , which the CSI driver expects.
I would say that's a wrong usage for zone.

following config would create a pv that could only be scheduled to a non-zone node with fault domain , if there is no node supports zone, then should not be set.
But nodes report that they are in , therefore it make sense to accept the same input in StorageClass.
if we translation "0" to "", then the pv created by that sc could be scheduled to any non zone node with any fault domain. That's a corner case that cluster is not in zone, and user still wants to set in sc
I am not sure I understand. If I provision a CSI volume in the westus cluster, its PV gets anyway: I have the cluster only with nodes in "westus" region, so all of them are without any zone. What would the CSI driver do if there were nodes in "westus" and say "ukwest", which does not have zones either? I can't test that setup easily, but I think the driver would report for both of them anyway, hence translating to does not make things worse, I think.
/sig storage /sig cloud-provider /area provider/azure
can you please check my previous comment? In what cases would be translating "0" to "" worse than the translation code we have today?
it's not worse or better, a single num () is actually not a valid zone, such in sc is actually not valid.
I mean we could translate the single digit to to make it work while the usage is wrong.
/sig storage
/sig cloud-provider


Question: t.Errorf("expect to get %d, but got %d", max-1, f) } } <ins> func TestAllocateUniqueness(t *testing.T) { max := 128 dynamicOffset := 16 uniqueAllocated := map[int]bool{} m := NewAllocationMapWithOffset(max, "test", dynamicOffset) // Allocate all the values in both the dynamic and reserved blocks for i := 0; i < max; i++ { alloc, ok, _ := m.AllocateNext() if !ok { t.Fatalf("unexpected error") } if _, ok := uniqueAllocated[alloc]; ok { t.Fatalf("unexpected allocated value %d", alloc) } else { uniqueAllocated[alloc] = true } } if max != len(uniqueAllocated) { t.Errorf("expect to get %d, but got %d", max, len(uniqueAllocated)) } } </ins>

In a enabled cluster, Kubernetes assigns duplicate IP addresses when the dynamic block is exhausted. I expected all clusterIPs are uniquely assigned. a enabled Kubernetes v1.24 cluster with a small service-cluster-ip-range. I confirmed on the minikube with the following command. services until the dynamic block is exhausted. see some clusterIPs are duplicated. No response <details</details<detailsminikube v1.25.2 </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
When the dynamic block is exhausted, we expect the service creation to fail with an error message like "No IP address available", right?
According to , Dynamically IP assignment will use the upper band by default, once this has been exhausted it will use the lower range. So I think using the reserved block (the lower range) is expected behavior
/sig network
upps /assign
/assign great catch, thanks


Question: reporter.SendUpdates() } <ins> func (reporter *ProgressReporter) SetTestsTotal(totalSpec int) { reporter.TestsTotal = totalSpec reporter.SendUpdates() } </ins> // ProcessSpecReport summarizes the report state and sends the state to the configured endpoint if set. func (reporter *ProgressReporter) ProcessSpecReport(report ginkgo.SpecReport) { testName := strings.Join(report.ContainerHierarchyTexts, " ")

Our team uses sonobuoy to run the conformormance tests on a cluster and displays the results to the users. We have been using conformance image 1.21.x until now and it sends back progress data to sonobuoy pod and our service reads it from the sonobuoy pod. The progress data included the total number of tests to be run and the completed tests. For 1.21 conformance we were receiving this data properly but when switching to 1.25.x conformance we are getting the total tests as zero. This is causing issues in our UI. Not sure what version of conformance image introduced this but we were using 1.21 and started seeing this issue after moving to 1.25.x. The conformance image versions in between 1.21 -1.25 are not CVE free so we cannot use them. Total number of tests should be non-zero for conformance image 1.25.x. Ideally i think it should come back as 360. provision a 1.25 kubernetes kind (or anything else) cluster Install sonobuoy Run conformance scan on the cluster Check the annotation on the sonobuoy pod . This is set by sonobuoy based on the data it gets back from the e2e conformance pod. The total tests comes as zero I just found it on 1.25.x conformance. Not really sure when this was introduced. Need to make sure back dated versiosn also have this fix in case they have this issue. <details25 cluster with 1.25.x conformance with any sonobuoy version </details<detailsANY </details<details</details<details</details<details</details<detailsConformacne e2e plugin </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
FYI
Yeah, it seems that it gets the info here I vaguely remember something about a progress custom reporter that was by sonobuoy and we didn't know what was used for
You remember correctly: It looks like that TODO didn't get addressed? I thought it had been. the other thing that was unclear regarding sonobuoy was whether it reads these test statistics from stderr or gets them via HTTP. We removed the stderr output (I think in 1.26) but kept (). This seems to be backed by (sonobuoy sets the parameter), but it would be good to get definite confirmation. Also, does Sonobuoy run tests in parallel? If it does, then these statistics are probably useless or at least will need considerable post-processing.
We depend on for this.
/assign I will take a look as has been fixed and closed
/reopen The fix went in, but let's keep the issue open for a while longer until we really know that it now works as expected. /assign
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle rotten [1]:
/remove-lifecycle rotten I discussed this further with (see also ) and while a final decision is pending, I am inclined to remove the progress reporting feature because I suspect that it is conceptually broken (to be confirmed) and lacks in-tree support (unit tests, owner). Before I do that, can you clarify: Are you running tests in parallel and does progress reporting work properly, i.e. does it go from 0 to 100%? When you said, "causing issues in our UI", what UI is that?
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
Ping Please see
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: finalAttached:        true, trigerWatchEventTime: 100 * time.Millisecond, timeout:              50 * time.Millisecond, <del> sleepTime:            5 * time.Millisecond, </del> shouldFail:           true, }, } for i, tc := range testCases { <del> fakeWatcher.Reset() </del> <ins> plug, fakeWatcher, tmpDir := newTestWatchPlugin(t) defer os.RemoveAll(tmpDir) attacher, err := plug.NewAttacher() if err != nil { t.Fatalf("failed to create new attacher: %v", err) } csiAttacher := attacher.(*csiAttacher) </ins> t.Logf("running test: %v", tc.name) pvName := fmt.Sprintf("test-pv-%d", i) volID := fmt.Sprintf("test-vol-%d", i)

/kind bug /sig storage What happened: CSI unit tests occasionally end up with data race: How to reproduce it (as minimally and precisely as possible): to get CSI unit tests into a binary: the binary 500 times in parallel until it breaks: . Use different argument to really load your machine, it should not be idle! . On my Linux, I got 3 failures in ~1000 runs. puts stderr of failed runs into .
/assign
I make a Fix: PTAL


Question: indexers *cache.Indexers, objectType reflect.Type) *watchCache { wc := &watchCache{ <del> capacity:            defaultLowerBoundCapacity, keyFunc:             keyFunc, getAttrsFunc:        getAttrsFunc, cache:               make([]*watchCacheEvent, defaultLowerBoundCapacity), lowerBoundCapacity:  defaultLowerBoundCapacity, upperBoundCapacity:  defaultUpperBoundCapacity, </del> <ins> capacity:     capacity, keyFunc:      keyFunc, getAttrsFunc: getAttrsFunc, cache:        make([]*watchCacheEvent, capacity), // TODO get rid of them once we stop passing capacity as a parameter to watch cache. lowerBoundCapacity:  min(capacity, defaultLowerBoundCapacity), upperBoundCapacity:  max(capacity, defaultUpperBoundCapacity), </ins> startIndex:          0, endIndex:            0, store:               cache.NewIndexer(storeElementKey, storeElementIndexers(indexers)),

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /cc /priority critical-urgent /milestone v1.19
/sig api-machinery
Suspect it has to do with cc /assign (for CI signal ownership)
Seems unlikely since multiple kind e2e tests passed on the PR itself
I was thinking that as well, but I don't see any other relevant changes
Actually, I take that back. Looks like the same failure was encountered as a flake on some of the PR runs. Looking at the test, it appears to assume the resource version of the kubernetes service will always be in the watch window, which is a bad assumption. I'd expect us to fix that assumption in the test, but also make sure we didn't unintentionally shrink the watch window for resources.
/assign
Note: this also appears to be breaking
Failure seems to be timeout related as well shows the runtime ~tripled after merged
revert open in can investigate test failure and timeouts and bring the change back in
I take it back, the test tolerates an expired error explicitly.
showed 17 of the 22 runs since merged took ~1 hour. The first run after the revert at returned to less than 30 minutes for the run. Will keep an eye on that to see if subsequent runs match.
- thanks a lot for looking into it. I don't think my PR changed anything that anyone should rely on. I will take a deeper look into it and try to fix that (hopefully later this week).
contains the explanation and describes the bug (in fact this bug still exists in the codebase, that PR was only increasing the probability of this particular test hitting it)