Question: if !ok { return nil, ErrFull } <del> return utilnet.AddIPOffset(r.base, offset), nil </del> <ins> return addIPOffset(r.base, offset), nil </ins> } // Release releases the IP back to the pool. Releasing an

What happened: All the services are asking to be recreated because the service CIDR changed after upgrading from 1.16.3 ro 1.17.0 What you expected to happen: Smooth migration from to , which would not break service CIDR. How to reproduce it (as minimally and precisely as possible): Do an upgrade from version 1.16.3 to 1.17.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
do you mean that the service cidr changed without any intervention after the upgrade? EDIT 96.0.0/12 is the default service cidr when using kubeadm --service-cidr string Default: "10.96.0.0/12" EDIT if not using kubeadm the default is May it be possible you were using another service cidr before the upgrade and you didn't set it during the upgrade, so it fall back to the default?
Can you provide your kube-apiserver invocation?
xref which warnings if no explicit CIDR flag was passed but should not have changed behavior otherwise
Where do you see that? The default is specified here as 10.0.0.0/24 and did not change between 1.16.0 and 1.17.0:
I was assuming it was using kubeadm based on the log message 96.0.0/12 is the default service CIDR for kubeadm
ah, ok. If they are using kubeadm, that default also did not change between 1.16.0 and 1.17.0:
a few questions for you: how did you install your 1.16.x cluster? did you customize the service cidr in that deployment? how did you upgrade your 1.16.x cluster to 1.17.x? did you apply the same service cidr customization in that deployment?
16.x was a straight and normal install no change on service CIRD, and we have never touched this CIDR configuration. And kubeadm is supposed to be used during the upgrade from 1.16.x to 1.17.x.

We just followed the describe process: The logs system started showing the shared message.
can you reproduce?
can you paste ? That will show you kubeadm config
created a single node cluster with v1.16.4 and a custom service cidr: applied upgrade to v1.17.0: i do not see a change in the custom service cidr.
if you never touched the service CIDR on the initial install it should have been which is the kubeadm default. your config map shows . how did you adapt the IP to work with the kubeadm default service cidr () if you initially deployed without modifying said cidr? to me this suggests that a change was required at some point.
The only change is the update from 1.16.3 to 1.17.0, done yesterday. Please find below a snapshot from our logs: <img width="1635" alt="Screen Shot 2019-12-21 at 22 22 29" src="https://user-"
my point is that if you did not apply a custom service cidr on "kubeadm init", then this cidr was the default out of kubeadm and after upgrade the service cidr was preserved as your config map shows. what was the elasticsearch/logging service IP before upgrade? EDIT: also do you have the YAML that was used to create this service?
101.249.43 belongs to the service rubnet range at minimum the error message is wrong /kind bug
are those errors happening constantly or only happened during the upgrade? are you able to create new services within that range? let's say, with a service IP of 10.101.249.45 Also, can you paste your current service-cluster-cidr configured in the controller-manager and apiserver? i.e.
this is a bug in core: this call in the repair controller returns for an IP that is actually in the CIDR range: also indicates a missing unit test case. latest commits: this might have regressed with the dual-stack changes? cc /remove-triage support needs-information /sig network /remove-sig cluster-lifecycle
/priority important-longterm
I was able to reproduce the same behavior again, with a fresh install k8s release 1.16.4.

Just After installing metrics server, we could notice a shift in the service CIDR, but nothing in the logs.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/cc
so the problem here is that an IPv4 CIDR such as translates to addresses. but this line here clamps the maximum to 2^16 () so for an address like the logic in the ip allocator fails in the function: as the offset ends up bigger than the maximum. it feels like the maximums for IPv4 and IPv6 should be different and larger. otherwise for CIDRs in the notation : can give incorrect results for IPv4 can give incorrect results for IPv6 cc who touched related logic.
I think that was touched in … was this bug preexisting or did that change behavior?
before PR () i do not see a clamp for IPv4: only one for IPv6 at 16bits.
We should: revert add unit tests for the missed cases get that into 1.17.1 reintroduce the reliance on the function carefully
What would be the safest path to recreate all the services whilst waiting a final solution ? Especially for production K8S.
to mitigate the error in 1.17.0 you need to modify the serivce CIDR to be , instead of . modify the ConfigMap CIDR to have modify the kubeadm static pod manifest for the api server () to have edit problematic Service objects and give them an IP in the range if needed.
Thank you for those instructions, I'm affected by this, does this have anything to do with a possibility of ClusterIP's not properly responding or does it only print out alerts?
hello, can you test if the ClusterIP is not responding? i don't think we got a confirmation if this is only an alert. my guess is that it is just that, but there might be side effects.
i see you logged i will comment there.
How do you "recreate" a service? Literally delete and create again? Or is there a command which would cause it to re-create itself without having to be deleted?


Question: --etcd-servers="http://${ETCD_HOST}:${ETCD_PORT}"  --advertise-address="10.10.10.10"  --cert-dir="${TMP_DIR}/certs"  <del> --feature-gates=AllAlpha=true,OpenAPIEnums=false  </del> <ins> --feature-gates=AllAlpha=true,OpenAPIEnums=false,ConsistentListFromCache=false  </ins> --runtime-config="api/all=true"  --token-auth-file="${TMP_DIR}/tokenauth.csv"  --authorization-mode=RBAC

Verify Master - Master Blocking verify.openapi-spec Inconsistently since 2 November. First failure (shown in testgrid) API Server is not becoming ready during Relevant logs /sig-testing
cc for cidr controller
and may be related.
/sig auth network
/remove-sig auth Is almost certainly related
/assign it seems the servicecidr controller takes time to start and it blocks the apiserver readiness
/reopen Test run failed after related PR merger:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yeah /assign
/triage accepted
this is sig network can you pelase update the title?
/retitle [Flaking Test][sig-network] ci-kubernetes-verify-master verify-openapi-spec
This still flake in master blocking test grid.
use 60s timeout and here uses 30s timeout by default. It flakes. 10 lines of log cannot show other details. My local test with 15s timeout failed several times. 30s is enough for my local tests. Or we just enlarge the timeout for CI.
having both scripts the same timeout sounds reasonable I'm still worried why it keeps timing out, but for that we should first define a test that measure the apiserver startup and then decide what value is correct and enforce it.
/priority important-soon /milestone v1.30 backport to v1.29 may be needed.
Hello! Release Signal shadow here. This issue has not been updated for a long time, so I'd like to check what's the status. The code freeze is starting (less than a month from now), and while there is still plenty of time, we want to ensure that each PR has a chance to be merged on time. As this issue is tagged for 1.30, is it still planned for this release?
This flakes in Do we have some action items for v1.30?
I need to fix it, is a shame on me, sorry
Hello! Release Signal shadow here. This issue has not been updated for a long time, so we'd like to check the status. The code freeze is starting less than a week, and we want to ensure that each PR has a chance to be merged on time.

As this issue is tagged for 1.30, is it still planned for this release? Please let us know the status if possible.
yeah, I like to get it fixed, there is some race condition that makes some of the boostrap to deadlock , I still didn't find why
didn't forget but didn't have time to figure out the root cause, will try to get back to this end of the week
I got one failure locally of an apiserver that took much time The apiservers need the servicecidr controller to start to be healthy, and the controllers need to sync the informer, but checking one of the requests from the informer I can see this error logging error output: "k8sx00nfnx02v1x12x06Statusx12x96x01nx06nx00x12x00x1ax00x12aFailurex1a2Timeout: Too large resource version: 2, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" and this keep going for a while logging error output: "k8sx00nfnx02v1x12x06Statusx12x98x01nx06nx00x12x00x1ax00x12aFailurex1a4Timeout: Too large resource version: 203, current: 1"aTimeoutCnx00x12x00x1ax00"7nx17ResourceVersionTooLargex12x1aToo large resource versionx1ax00(x012x000xf8x03x1ax00"x00" do you know why the resource version is growing?
looks like the server returns that error in a few places, not sure which one is getting exercised: update-openapi-spec starts with AllAlpha=true in order to enable all features / APIs ... are there any alpha features around watch / resourceVersion that could cause this?
Didn't read carefully the issue, so just quickly sharing some thoughts: 1) streaming lists might potentially be related, but it needs to be explicitly requested by the client.

Reflector is the only place that can currently do that, but it requires explicitly set env-var: Is it possible that this value is set? if so, then streaming lists will result is calling etcd for current RV and will wait for watchache to be synced with it - if progress-notify-interval is large and manual progress notify is disabled, we could face things like that ^^ 2) that said, if Alpha features are enabled, then manual-progress notify should also be set: 3) is it possible that we're facing the bug that Marek found 2 days before release that effectively was the main reason we didn't try to promote consistnet read to beta? ^^
Based on the log from I would guess this is the 3002ms implies the 3 second timeout of waiting for revision Empty implies consistent list from cache implies non-recursive list Planned to fix it, but PR was blocked on
thanks a ton for digging into that to deflake this test, I'd set this in the update-openapi- file (since it doesn't impact the generated openapi) with a comment to remove once is resolved


Question: return "", errors.New(ErrDeviceNotFound) } <ins> // cleanPath remove any trailing substrings that are not part of the backing file path. func cleanBackingFilePath(path string) string { // If the block device was deleted, the path will contain a "(deleted)" suffix path = strings.TrimSpace(path) path = strings.TrimSuffix(path, "(deleted)") return strings.TrimSpace(path) } </ins> // FindGlobalMapPathUUIDFromPod finds {pod uuid} bind mount under globalMapPath // corresponding to map path symlink, and then return global map path with pod uuid. // (See pkg/volume/volume.go for details on a global map path and a pod device map path.)

A pod uses a PVC with . The block volume is backed by a block device on the node (which is created by some CSI storage providers). If the underlying block device corresponding to the volume is accidentally disconnected from the node (due to network, hardware problem, etc...), the user cannot delete the pod. The pod would be stuck in terminating forever since Kubelet refuses to unmount the block volume. This prevents the user from cleaning up the pod and spinning up a new replacement pod thus leading to a long service degradation. For example, if the pod is part of a statefulset, the replacement pod cannot come up due to the old pod being stuck terminating. Kubelet should be able to finish terminating a pod when the corresponding block device is already removed from the node. a CSI storage provider that supports . For example, a statefulset that has volumeClaimTemplates with . For example: <details close<summaryclick to open yaml manifest</summary</detailsa network/hardware problem in which the backing block device is removed from the host unexpectedly. In our example, we delete a Longhorn engine instance manager which leads to the underlying block device being removed from the host. that the pod cannot access the block volume, try to delete the pod. pod is stuck in terminating state forever. Logs from kubelet: <details close<summaryClick to open logs</summary<br</detailsPlease pay attention to the log . Kubelet is refusing to terminate the pod because the underlying block device is removed from the host. If we run on the host, we can see that the backing file of the corresponding loop device is marked as : In our humble opinion, we think that Kubelet should process to remove the loop device corresponding to the deleted block device so that pod can finish terminating as detailed in this PR: We have tested that PR and can confirm that it fixes the issue. We have prepared a PR at : We have tested that PR and can confirm that it fixes the issue. <details</details<detailsRancher RKE2 </details<details</details<detailsRancher RKE2 </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance.

The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Ref:
/cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle rotten
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
kubelet log 13:31:57 wanguo-72 kubelet[1131]: E0428 13:31:57. 1131 ] Operation for "{^-54fa-4068-bbad- podName:-9521-4aed-9bfe- nodeName:}" failed. No retries permitted until 2023-04-28 13:33:59. +0800 CST m=+8838. (durationBeforeRetry 2m2s). Error: UnmapVolume.UnmapBlockVolume failed for volume "pvc--a4e4-48a7-9057-" (UniqueName: "^-54fa-4068-bbad-") pod "-9521-4aed-9bfe-" (UID: "-9521-4aed-9bfe-") : blkUtil.DetachFileDevice failed. globalUnmapPath:, podUID: -9521-4aed-9bfe-, bindMount: true: failed to unmap device from map path. mapPath is empty node poweroff restart , pod use block mode volume, and pod stuck in terminating
so it is a new issue, and I encounter it as well with XSKY CSI after node restart. My kubernetes version is 1.25.8
which Kubernetes version is it?
Hi, I also see the issue reported by I'm using v1.25.10. I assume it happens when using containerized kubelet. In my environment, losetup reports weird backing file paths. This can be reproduced as follows. Until this moment, the correct path is visible from the host. But, after terminating the container, I get the wrong results. I consider this behavior causes failing unmap. Any thoughts?


Question: }, }, }, <ins> Labels: map[string]string{"a": "b"}, </ins> } func TestCreateController(t *testing.T) {

If a ReplicaController has a selector that doesn't match the template it'll just create tons and tons of pods. It should validate that the pods it is creating have a good chance of satisfying its selector.
It would also be useful to validate that the selector doesn't obviously overlap with selectors of other replicationControllers.


Question: } // metricsCollector collects metrics from legacyregistry.DefaultGatherer.Gather() endpoint. <del> // Currently only Histrogram metrics are supported. </del> <ins> // Currently only Histogram metrics are supported. </ins> type metricsCollector struct { *metricsCollectorConfig labels map[string]string

The new scheduler performance test was introduced back in 1.18, back then, the metric it supported are all . However, along with , some metrics are updated to be , and hence it's no longer appropriate to return the first Histogram: However, it's very common (for instance Preemption) that the metrics may hold multiple histograms: Additionally, metrics and have been removed, we should update them to : /sig scheduling
/triage accepted /assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: "sessionTicketKeys", ) <ins> // See #33936. if strings.HasPrefix(runtime.Version(), "go1.7") { expected.Insert("DynamicRecordSizingDisabled", "Renegotiation") } </ins> fields := sets.NewString() structType := reflect.TypeOf(tls.Config{}) for i := 0; i < structType.NumField(); i++ {

When using go 1.7, is failing with the following error:
Emm... I'll need this fix
Wouldn't a full struct copy work if we zero out the fields we don't want retained afterwards? It'll be less fragile as well.
You prefer to re-open my pull request or include that commit in ?
I'm also trouble in the issue when I execute command on release-1.4 branch of Kubernetes. Have someone fixed it? Fails (all fails in ):
, we can not just re-open : in , it did not include & which means build failed in go1.6 if re-open .
BTW, do we still support go1.6 in master? If so, we need upmerge to master.
Is this still a problem? I don't see any test failures on go-1.7.3

Hey Guys, When running command on release-1.4 branch, I got the below errors, my GO version is , how to fix it or something else have I missed?
dup of . fixed by . we can close this. /sig api-machinery
Sure, close it.


Question: - containerPort: 443 name: https protocol: TCP <ins> readinessProbe: httpGet: path: /readyz port: https scheme: HTTPS periodSeconds: 10 failureThreshold: 3 livenessProbe: httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 failureThreshold: 3 </ins> volumeMounts: - mountPath: /tmp name: tmp-dir

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- (gce-ubuntu-master-default) Kubernetes e2e suite: BeforeSuite (There are other flakes in this job, but they are failing less often and different test each time, so let's say this issue is only for BeforeSuite) is not starting: Spyglass: Triage: Mentioned in /cc <!-- Please see the deflaking doc () for more guidance! --
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc Since it is related to the metrics server. the metrics server keeps dying as the log shown in I cannot get more logs of metrics servers from the CI artifacts. Currently, we are using metric server 0.4.4 and the latest 0.5.0 is released. Are there some known bugs that may be related to this test failure?
I don't think so. /cc
The metrics-server fails to bind ~because there is another process listening~ on port 443 are some of the konnectivity pods listening on the port :443?
In the dockerfile of metrics-server if has ? Maybe it has permission problem?
ahh, right, nice catch, I didn't pay attention to the error and blindly assumed collision :sweat_smile: ... who can take an action on that? has this changed recently?
If the metrics-server version used after , I think it is correct. has no change recently.

I vaguely remember some changes recently to reduce privileges on pods on CI, tthis seems likely a permission problem, does this ring a bell? EDIT This PR introduced the change , any special reason why we want to run it in a privileged port if this is accessed using a Service? EDIT 2 let see if this is the problem
Yeah, Maybe you can refer to this, the release notes about v0.4.4 metrics-server . I think you have resolved this problem.
Release v0.3.X was already using privileged port 443, as v0.4.X has migrated to distroless image we needed to introduce to give it needed privileges. What's weird for me is that the test is flaking and not failing. Could it be related to node where MS is scheduled?
You mean, there may be other reasons? If so, may require in-depth analysis.
As far as I can tell, is never starting properly, even when the jobs are successful. If we were to take this job, that has succeeded: From the serial logs, we can see that the metrics-server pod is crash looping: But from the build logs, we can see that at some point the pod was deemed ready: To me it seems that there are 2 issues, one being that metrics-server doesn't have the capabilities it needs to run and another one that still needs to be investigated where metrics-server is sometimes marked ready even though it is in a crash looping state.
This is a problem with the metrics-server deployment on the CI jobs and the metrics tests I don't know which sig owns these components but it will be better to find first an owner, what is is clear is that the metrics tests are not reliable since once you set a readiness probe on the metrics-server it fails consistently , , however the tests pass cc
it seems the last try was the good one :smile:


Question: } if len(t.BearerToken) == 0 { var err error <del> t.BearerToken, err = generateSecureToken(16) </del> <ins> t.BearerToken, err = GenerateSecureToken(16) </ins> if err != nil { klog.Fatalf("Failed to generate bearer token: %v", err) } } // Allow 1% of nodes to be unready (statistically) - relevant for large clusters. if t.AllowedNotReadyNodes == 0 { t.AllowedNotReadyNodes = t.CloudConfig.NumNodes / 100

Which jobs are failing: Which test(s) are failing: The tests are failing in the . Since when has it been failing: 10/23/2020 Testgrid link: Reason for failure: It started failing at commit , but it's unknown if it's related to that. Anything else we need to know: We discussed in the Kubernetes SIG-Node CI subgroup and will start taking a look at it.
/assign
/sig node
Investigation notes: between successful and unsuccessful runs and found no notable differences. , and had a couple of failed health checks on initialization, which doesn't happen in the successful test runs, though there's only 2 seconds between starting readiness checks and an eventual successful check. config the same between runs looking through the kubelet logs for and , but I haven't found anything interesting yet. kubelet- failed health checks The below is a list of errors that occur in runs that succeeded and failed, which may be due to startup or specific testing scenarios. Bunch of these for numerous containers:
Both new and old test runs have a bunch of logs like this, implying that it's been having issues for a while: The failed one has this additional line: However, the doesn't have the same issues. I don't see the connection timeouts at all. I also don't see anything else in the kubelet logs that appears different between successful and failed runs. Unfortunately, I don't know enough about the infrastructure setup, but here are my current theories: about the infrastructure has gotten progressively more overloaded and now it's falling over. The tests run on 8 nodes versus 1, so they have more capacity there. Not totally buying this because why did it go from mostly successful to 100% failures rather than a mix for a bit. changed in the GCE configuration. Not sure how that is managed.
Diff there suggests three merges from k/k Only merge in test-infra is an image bump <img width="215" alt="image" src="https://user-"Interesting that a similar ci version and pull version of the job passes and the one ci version does not. pull: passing ci: failing ci: The failing ones do not have a node_e2e , but I don't see any of the expected log messages for failure of the services to start.
It is interesting that one passes and one does not.

Primary differences are the number of nodes, resource settings, and the fact that the conformance tests use Docker.
In addition to the services log missing, the overall system run logs are also missing. May be related to any decision we make for
Would we be able to kick off a Prow job with custom configuration? In other words, to rule in or rule out something in the image change, can we kick off a job with the latest everything else but use the older image? In reference to the decision on , are you suggesting we pause investigation and see if we decide to just move or change the conformance tests?
/triage accepted /priority critical-urgent
Catching up on this. qq: has anyone vetted ? Ise that pr went in around the time this job began to fail The beforesuite (AI: we should list out what it does), fails to communicate with nodes the mention of 6443 ports looks interesting here.
Good find. Definitely looks suspicious. I'll dig into that.
From : This comes from , which has some node-conformance specific test setup, so maybe something interesting there.
quick interruption for instructions on how to reproduce this one: I think this may work Focus and skip The FOCUS and SKIP arguments im getting from Image config file The job also uses the following config file Make sure to copy it and name it (the argument to IMAGECONFIG_FILE above). Additional arguments The test args to this job are I think these are the defaults already but just in case may be actually useful to add the following to the above command Miscellaneous If you run into trouble with the above commands try going through The info here came from
You'll also need to use the flag to make sure it runs in Docker, I believe.
ah yeah, good catch! Thank you. Don't know about your availability, but if you are all busy ima allocate sometime later today to dive deeper into the info we have here
I'm going to try to spend time of this in the afternoon (PDT). Specifically, I was going to try to verify that the logs from starting up the test suite look as expected and possibly try to run the case to reproduce.

My first suspicion was to have the server running in another port than 6443 (which Kubelet has configured), this is not the case: Only o.SecureServing.BindAddress is being set explicitly, but the port comes from o.NewServerRunOptions().
I think it's related to the movement of this line. I couldn't figure out how to link to the PR so I made a comment.
as you notice before there's no , and we should be seeing Before:
I can repro the issue now, and I think the line was moved upward to generate the token before starting the kubelet. Moving it up out of the if-statement may help partially, but in the conformance test the kubelet is manually started ahead of this code, so it may not matter.
Make sense. Do you mean moving this out of the conformance check?
Haha, yea, that message is deceptive... I think I'll make a PR to at least let it be more correct and give us more info.
for alerts to subscribers, PR created. Please comment.
looking green!
The output looks very strange, I have no idea why. Output has double-underscore-number for the different runs, instead of the image used. [__08] vs [cos-lts]
Ya, I noticed that. It's weird. Not sure how it's set, but you can see that the junit config names have a similar issue: -junitcos- -
Ok, found the root of the problem. Here's the command for the master suite: Note , which is not present in the conformance tests. but not sure how it eventually gets here.


Question: fRC.mutex.Lock() defer fRC.mutex.Unlock() <ins> fRC.Calls = append(fRC.Calls, "Delete") </ins> if routes, ok := fRC.FakeStore[routeTableName]; ok { if _, ok := routes[routeName]; ok { delete(routes, routeName)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubernetes shouldn't put route table when it's already been configured. Azure cloud provider should check whether the route has already been created before invoking . What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /priority important-soon
Why this is a bug? A PUT operation should be idempotent and multiple calls should get the same result?
Right. But cloud provider shouldn't make unnecessary API calls.
To check whether it is configured, we need a GET call, so we're going to replace PUT call with GET call in this case?
it's not a replace, we need to LIST routes and check whether it has already been there, then PUT if it's not.
I got your point. But if we directly send a PUT request, seems route RP will also do this comparation? Is there any issue making us have to do the comparation by ourselves instead of sending PUT request to route RP?
It would help to reduce the chances of ARM PUT throttling.
I'd like to work on this
/assign


Question: # github.com/cpuguy83/go-md2man/v2 v2.0.2 ## explicit; go 1.11 github.com/cpuguy83/go-md2man/v2/md2man <del> # github.com/cyphar/filepath-securejoin v0.2.3 </del> <ins> # github.com/cyphar/filepath-securejoin v0.2.4 </ins> ## explicit; go 1.13 github.com/cyphar/filepath-securejoin # github.com/davecgh/go-spew v1.1.1

One of the dependencies of k8s, filepath-securejoin, has a following vulnerability discovered: It has been patched in version 0.2.4, however k8s is still using version 0.2.3. Potential fix: updating the dependency. A small PR will follow. /sig security
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig security
/kind bug /sig security


Question: windows/amd64 ) <ins> # Gigabytes desired for parallel platform builds. 8 is fairly # arbitrary, but is a reasonable splitting point for 2015 # laptops-versus-not. readonly KUBE_PARALLEL_BUILD_MEMORY=8 </ins> readonly KUBE_ALL_TARGETS=( "${KUBE_SERVER_TARGETS[@]}" "${KUBE_CLIENT_TARGETS[@]}"

Filing for This code is wrong The local needs to be on a separate line from the command execution, and we need to be resilient to not existing on OS X, like:
For once I was actually watching the output of a build and noticed that I was seeing an error about md5sum not being found. I'm building on a mac using boot2docker. The short_hash function () works fine on my mac as md5 does exist.
Actually, I'm just going to fix this as a follow-on to .
Thank you!


Question: ADDON_MANAGER_LEADER_ELECTION: $(yaml-quote ${ADDON_MANAGER_LEADER_ELECTION}) EOF fi <ins> if [ -n "${API_SERVER_TEST_LOG_LEVEL:-}" ]; then cat >>$file <<EOF API_SERVER_TEST_LOG_LEVEL: $(yaml-quote ${API_SERVER_TEST_LOG_LEVEL}) EOF fi </ins> else # Node-only env vars.

After we moved logging all requests in apiserver to V(3) level: We have the: --vmodule=httplog=3 to config of all scalability tests, because debugging scalability tests is pretty much impossible without having those logs. However, apparently this is adding a huge lock contention on that, because if vmodule is non-empty, we enter a non-trivial code-path in the glog for every single glog call: I did some experiments, and switching --v=3 for apiserver (instead of --v=2 + vmodule) doesn't introduce the significiant change - the difference in amount of produced logs is about 10% (a bit less). So given that we don't do that in production anyway, what we should do is to: set --v=3 for apiserver in our scalability tests (to have all the logs we need) get rid of those vmodule settings
Something went wrong here. The vmodule flag change worked as expected, but setting --v=3 in apiserver didn't. Here are the api-server args from the recent run of the gce scale perormance ! The vmodule is not set, but the log level is set to v2. As the result we currently don't have the http logs. I'm investigating why log verbosity is not overridden in api server.
Env variables seem to be set correctly in the build file: !
/reopen
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
seems to be fixing the issue - !


Question: o := NewExplainOptions(parent, streams) cmd := &cobra.Command{ <del> Use:                   "explain RESOURCE", </del> <ins> Use:                   "explain TYPE [--recursive=FALSE|TRUE] [--api-version=api-version-group] [--output=plaintext|plaintext-openapiv2]", </ins> DisableFlagsInUseLine: true, Short:                 i18n.T("Get documentation for a resource"), Long:                  explainLong + "nn" + cmdutil.SuggestAPIResources(parent),

Especially the options description are super sad. I think we can improve them in the next beta iteration cc /sig cli
/triage accepted
Hi does fixes this issue?
I agree with the same. Unlike other outputs, it doesn't share the details about the values or their usage.
Unfortunately, I don't think it does. Have you tried it? Please send a pull-request I'll happily review it.
Hi , I'd like to work on this. However, I would need some guidance so I know I'm in the right direction. My guess is we change the text I have noticed these descriptions are not as per the other flag descriptions in kubectl. I'll make a draft once you confirm.
For eg these are options in : we can try adding details and formal tone to the text. /assign
I don't know if you need my help, you seem to be doing just fine. Please go ahead and ask me questions if you have specific problems, I'll be happy to guide. Thanks!
That's correct.
sure thing , I'll send the PR
Hi How does this output looks :
Hi Raised the PR: for reviews on same.


Question: if printContainers { printValue(w, PodColumn) } <ins> sort.Slice(metrics, func(i, j int) bool { if withNamespace && metrics[i].Namespace != metrics[j].Namespace { return metrics[i].Namespace < metrics[j].Namespace } return metrics[i].Name < metrics[j].Name }) </ins> printColumnNames(w, PodColumns) for _, m := range metrics { err := printSinglePodMetrics(w, &m, printContainers, withNamespace)

and node should sort by namespace / name so that results don't jump around. We should consider adding sort, but only after we make a stable sort the default.
next Also, why does it sometimes lose nodes? Annoying
I did not reproduce this. I think this may be caused by the monitoring timeout of heapster. So that the data got missing.


Question: return ioutil.WriteFile(destFile, b, 0644) } <ins> type labelValues struct { label  string values []string } </ins> // metricsCollectorConfig is the config to be marshalled to YAML config file. <ins> // NOTE: The mapping here means only one filter is supported, either value in the list of `values` is able to be collected. </ins> type metricsCollectorConfig struct { <del> Metrics []string </del> <ins> Metrics map[string]*labelValues </ins> } // metricsCollector collects metrics from legacyregistry.DefaultGatherer.Gather() endpoint.

<!-- Please only use this template for submitting enhancement requests --What would you like to be : run got the below error message, We should provide the metric data for different extension points from the metric of "schedulerframeworkextensionpointdurationseconds" Why is this needed: Those metrics have been removed by this commit - Those metrics are deprecated cause it is suggested to use - , but simply remove those metrics definitely is not a nice way to address the issue. /sig scheduling
/triage accept
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /triage accept Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale


Question: Account mount path MUST be auto mounted to the Container. */ framework.ConformanceIt("should mount an API token into pods ", func() { <del> var tokenContent string </del> var rootCAContent string <ins> sa, err := f.ClientSet.CoreV1().ServiceAccounts(f.Namespace.Name).Create(&v1.ServiceAccount{ObjectMeta: metav1.ObjectMeta{Name: "mount-test"}}) framework.ExpectNoError(err) </ins> // Standard get, update retry loop framework.ExpectNoError(wait.Poll(time.Millisecond*500, framework.ServiceAccountProvisionTimeout, func() (bool, error) { By("getting the auto-created API token") <del> sa, err := f.ClientSet.CoreV1().ServiceAccounts(f.Namespace.Name).Get("default", metav1.GetOptions{}) </del> <ins> sa, err := f.ClientSet.CoreV1().ServiceAccounts(f.Namespace.Name).Get("mount-test", metav1.GetOptions{}) </ins> if apierrors.IsNotFound(err) { <del> framework.Logf("default service account was not found") </del> <ins> framework.Logf("mount-test service account was not found") </ins> return false, nil } if err != nil { <del> framework.Logf("error getting default service account: %v", err) </del> <ins> framework.Logf("error getting mount-test service account: %v", err) </ins> return false, err } if len(sa.Secrets) == 0 { <del> framework.Logf("default service account has no secret references") </del> <ins> framework.Logf("mount-test service account has no secret references") </ins> return false, nil } for _, secretRef := range sa.Secrets {

https://k8s- /sig auth /kind failing-test
Adding a priority to this: /priority important-soon Please raise the priority if you consider this a release blocker. Given the test job, it normally wouldn't be.
It's a test issue. I'm hoping to fix this before 1.13 but I don't consider it a release blocker.


Question: <ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package apiserver import ( "testing" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/api/meta" "github.com/emicklei/go-restful" ) func TestScopeNamingGenerateLink(t *testing.T) { selfLinker := &setTestSelfLinker{ t:           t, expectedSet: "/api/v1beta3/namespaces/other/services/foo", name:        "foo", namespace:   "other", } s := scopeNaming{ meta.RESTScopeNamespace, selfLinker, "/api/v1beta3/namespaces/{namespace}/services/{name}", true, } service := &api.Service{ ObjectMeta: api.ObjectMeta{ Name:      "foo", Namespace: "other", }, TypeMeta: api.TypeMeta{ Kind: "Service", }, } _, _, err := s.GenerateLink(&restful.Request{}, service) if err != nil { t.Errorf("Unexpected error %v", err) } } </ins>

Accessing entities without namespaces in the url makes the selflink to look corrupted (has namespace missing): Example - invoking: http://localhost:8080/api/v1beta3/services/ returns entities with this selflink (note the part after namespaces) "selfLink": "/api/v1beta3/namespaces/%7Bnamespaces%7D/services/guestbook"
I thought accessing namespaced objects without the namespace was just disallowed in v1beta3. ?
Oh, probably list is allowed for components that need to watch all objects across all namespaces, such as kube-proxy. In that case, the selfLink looks broken, yes.
- right. I will look into the self link this afternoon to see what's going on. Self assigning.
The api has to allow fetching all objects cross namespaces without specifying namespace, otherwise it will cause multiple requests to the api - per namespace, big overhead for clients.


Question: glog.V(4).Infof("ensureExternalLoadBalancer(%s): Service needs local traffic health checks on: %d%s.", lbRefStr, healthCheckNodePort, path) if hcLocalTrafficExisting == nil { // This logic exists to detect a transition for non-OnlyLocal to OnlyLocal service <del> // turn on the tpNeedsUpdate flag to delete/recreate fwdrule/tpool updating the </del> <ins> // turn on the tpNeedsRecreation flag to delete/recreate fwdrule/tpool updating the </ins> // target pool to use local traffic health check. glog.V(2).Infof("ensureExternalLoadBalancer(%s): Updating from nodes health checks to local traffic health checks.", lbRefStr) if supportsNodesHealthCheck { hcToDelete = makeHttpHealthCheck(MakeNodesHealthCheckName(clusterID), GetNodesHealthCheckPath(), GetNodesHealthCheckPort()) } <del> tpNeedsUpdate = true </del> <ins> tpNeedsRecreation = true </ins> } hcToCreate = makeHttpHealthCheck(loadBalancerName, path, healthCheckNodePort) } else { glog.V(4).Infof("ensureExternalLoadBalancer(%s): Service needs nodes health checks.", lbRefStr) if hcLocalTrafficExisting != nil { // This logic exists to detect a transition from OnlyLocal to non-OnlyLocal service <del> // and turn on the tpNeedsUpdate flag to delete/recreate fwdrule/tpool updating the </del> <ins> // and turn on the tpNeedsRecreation flag to delete/recreate fwdrule/tpool updating the </ins> // target pool to use nodes health check. glog.V(2).Infof("ensureExternalLoadBalancer(%s): Updating from local traffic health checks to nodes health checks.", lbRefStr) hcToDelete = hcLocalTrafficExisting <del> tpNeedsUpdate = true </del> <ins> tpNeedsRecreation = true </ins> } if supportsNodesHealthCheck { hcToCreate = makeHttpHealthCheck(MakeNodesHealthCheckName(clusterID), GetNodesHealthCheckPath(), GetNodesHealthCheckPort())

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig network /area cloudprovider What happened: The cloud provider interface that should . The GCE provider does this for but not for . Every other in-tree provider implements by fully updating the load balancer including updated backend nodes: , , , And the following out-of-tree providers do as well and Oracle (not oss yet). What you expected to happen: I expect that all cloud providers implement by creating or fully updating the load balancer if it already exists. Related discussion: /cc
/assign Make sense to make implementation in GCE consistent with others for the ease of up-coming service controller enhancement. This sounds like a 1.10 task.
is this task blocking you on another attempt at refactoring the service controller sync logic?
yup


Question: <ins> { "kind": "ConfigMapList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/namespaces/edit-test/configmaps", "resourceVersion": "252" }, "items": [] } </ins>

Try running and adding an ingress object, or an empty list, then exit the editor – it will panic like this: I've also tried , and for the sake of experiment tried adding a Pod, then it failed rather gracefully with this message: But when I tried to add a DaemonSet, it paniced. I am not sure how we should fix, perhaps editing of lists isn't something a normal user is expected to do... Let's discuss.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig cli
So I wanted to test how it would behave if I take an item out of my list, so I've create and ingress object and tried . To my surprise I've found only one ingress object without a list. When I a second object, gave me a list to edit. So I must say that flattening of the list in case there is only one object was a bit surprising. Now that I have a list of two items, I've decided to try deleting one by removing it from the list, and it turns that's not possible.
It feels like the problem is complicated, and I'd rather see disallowed unless users passed a name of a specific resource. If we belive some folks rely on to work when there is only one resource to edit, maybe we can keep that... If we cannot disallow this, I'd rather see it fixed in a way where removing items results in deletion and editing a single item list is actually about editing a list. At the very least, of course, we should fix the panic, but I still want to hear what people think about this before I open a PR (presumably fixing that panic should be trivial).
I'm working on this. Already submitted a PR in . PTAL.
I think at the very beginning, at least v1.5.0 the logic is already like this, I'm not sure why this bothering you I use this lot of time, I'd prefer keep it.

also cc
I don't think edit should open with an empty list. should open a list if there are multiple items, a single item if there is only one, and error if there are none of that type. should open a single item if it is found, and error if it is not found
that sounds good to me. I was just a bit surprised, that's all.


Question: - user: root - name: /opt/cni - makedirs: True <del> - source: https://storage.googleapis.com/kubernetes-release/network-plugins/cni-8a936732094c0941e1543ef5d292a1f4fffa1ac5.tar.gz </del> <ins> - source: https://storage.googleapis.com/kubernetes-release/network-plugins/cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz </ins> - tar_options: v <del> - source_hash: md5=ae7ec24d2ffc0fd14a15f527744ba2c3 </del> <ins> - source_hash: md5=1ac2f8c72244f0188b8ba3d6052f9fc1 </ins> - archive_format: tar - if_missing: /opt/cni/bin

We should update cni binaries to containernetworking/cni and release binaries for all arches. I do not have push access to the repo, so I guess some googler have to do it.
Thanks. I have this on my plate.


Question: var _ registry.LogFormatFactory = Factory{} func (f Factory) Create(options config.FormatOptions) (logr.Logger, func()) { <ins> stderr := zapcore.Lock(os.Stderr) </ins> if options.JSON.SplitStream { <del> // stdout for info messages, stderr for errors. infoStream := zapcore.Lock(os.Stdout) </del> <ins> stdout := zapcore.Lock(os.Stdout) </ins> size := options.JSON.InfoBufferSize.Value() if size > 0 { // Prevent integer overflow. if size > 2*1024*1024*1024 { size = 2 * 1024 * 1024 * 1024 } <del> infoStream = &zapcore.BufferedWriteSyncer{ WS:   infoStream, </del> <ins> stdout = &zapcore.BufferedWriteSyncer{ WS:   stdout, </ins> Size: int(size), } } <del> return NewJSONLogger(infoStream, zapcore.Lock(os.Stderr)) </del> <ins> // stdout for info messages, stderr for errors. return NewJSONLogger(stdout, stderr) </ins> } <del> // The default is to write to stderr (same as in klog's text output, // doesn't get mixed with normal program output). out := zapcore.Lock(os.Stderr) return NewJSONLogger(out, out) } // writeWithFlushing is a wrapper around an output stream which flushes another // output stream before each write. type writeWithFlushing struct { zapcore.WriteSyncer other zapcore.WriteSyncer } func (f writeWithFlushing) Write(bs []byte) (int, error) { f.other.Sync() return f.WriteSyncer.Write(bs) </del> <ins> // Write info messages and errors to stderr to prevent mixing with normal program output. return NewJSONLogger(stderr, nil) </ins> }

When working on I have found a regression in JSON logging format that happend in 1.23. In v1.23 JSON logging introduced two new features, log stream splitting and info log buffering. When I started measuring throughput it looked very promising, for example for info logs showed 10x improvement when using JSON split stream and 100Mi info log buffer. However when I introduced mixed logs (ever X info logs write 1 error logs), I noticed a large regression. For 10% of error logs written (apiserver during upgrade can write even 30% errors), I have found that just switching json on, results in log throughput being reduced from 25 MiB/s to 0.7MiB. I have bisected git repo and found that was the root cause. I expect that this regression would be noticed if we run scalability tests with JSON enabled. Loader code: /wg structured-logging /cc /sig scalability JSON should generate similar or greater log throughput when compared to text format. Run loader as described in from No response N/A N/A N/A N/A N/A N/A
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So to reproduce the regression, I need to do (default for error rate is 10%, 300 for message length, 10s duration) for old and new code, right?
Yes, let me give my results. Before text: 30.7MiB/s, json:39.6MiB/s After, text: 30.6MiB/s, json: 6.5MiB/s
should we try to get this in 1.23.1?
Yes, we just moved JSON and Structured Logging to Beta in v1.23. This bug should cause performance issues when there is non trivial amount of errors (1%) written by control plane. I expect that this has really bad impact on upgrades which I have seen can reach 30% of error logs from apiserver.

It's worth pointing out that this regression only occurs when writing to a disk-based file (see ). That also provides a very likely explanation for the huge performance difference: I bet zap not only flushes its buffered info messages, it also syncs the file. That is fast for tmpfs, but slow for real disks.
do you know how the Kubernetes components are run in those cases where logging performance matters? Do they directly write to a real disk or does it all go through pipes? If it is the latter, then this issue matters a lot less in practice. you mentioned at some point that you had scalability issues with logging that went away when using go-runner as wrapper instead of using klog's internal output redirection. I bet the root cause is the same: fast vs. slow sync of the output fd. That's because go-runner always runs output through one or more pipes. I don't know how realistic the scalability setup was, but it must have written log files to some real disk.


Question: return err } wc.initialRev = getResp.Header.Revision for _, kv := range getResp.Kvs { <del> prevResp, err := wc.watcher.client.Get(wc.ctx, string(kv.Key), clientv3.WithRev(kv.ModRevision-1), clientv3.WithSerializable()) if err != nil { return err } var prevVal []byte if len(prevResp.Kvs) > 0 { prevVal = prevResp.Kvs[0].Value } wc.sendEvent(parseKV(kv, prevVal)) </del> <ins> wc.sendEvent(parseKV(kv)) </ins> } return nil }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): etcd watcher Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): HEAD / commit Environment: Cloud provider or hardware configuration: local VM OS (e.g. from /etc/os-release): Fedora 24 Kernel (e.g. ): Linux localhost.localdomain 4.7.7-200.fc24.x8664 SMP Sat Oct 8 00:21:59 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: N/A Others: N/A What happened: If you do a watch from resource version "0" using (the etcd v2 client code), it initially performs a get of the key in question and returns Added or Modified events, based on a comparison of each node's createdIndex and modifiedIndex. This works all the time. If you do a watch from resource version "0" using (the etcd v3 client code), it attempts to get for each kv returned, in an attempt to provide both the previous and current versions of each kv/object. This works as long as has not been compacted. If compaction has occurred, the initial will fail, and it's not possible to watch from "0"' for this particular key.

What you expected to happen: Either: v3 watcher should not change the behavior from the v2 watcher; namely, it should not attempt to get the previous revision of each object and instead just return the current version as a event agree that it's ok to change the behavior, and in the event that the database has been compacted and the previous revision of an object cannot be retrieved, that object is sent as a event instead of . How to reproduce it (as minimally and precisely as possible): I'm working on a unit test and fix. Anything else do we need to know: cc
I will take a look and get back to you tomorrow.
If you have a pr ready, just assign to me or hongchao.
I'll have one tonight hopefully
Although it depends on if we pick option 1 or 2
The behavior of etcd2 is not technically correct. Watch from 0 and will give different results if watch from 0 always return "Added" even if 0 and should be the same. But we do not really have defined what does "0" mean in the interface. Also for etcd3, we cannot guarantee that watch can always return the previous value. Option 1 seems to be easier to achieve. If we do option2, then we will have a "Modified" event without having previous value. I am not sure if that is acceptable or not.
Just give an update here: I talked with on this a bit. We agreed that the behavior should be the same as etcd2 watcher. Of course the API seems non-standard right now. We need to rewrite it in better API once this gets more stable.
Actually, it looks like the event type is a red herring.

The etcd2 client has always set Added/Modified based on the equality/inequality of the node's createdindex/modifiedindex: the watch cache always sets the event type to Added when watching from resourceVersion=0 etcd3 (when watching from 0) is still doing a Get() of the previous revision of every KeyValue: The extra get of the previous version is unnecessary if ModRevision == ModCreation, and can fail if the Get returns an ErrCompacted (it also seems like a potential performance problem to fan out N Get() calls during a single watch establishment
potential fix in
I pretty dislike the fact that: on 0 will first do a sync and then start to watch are forced to return prev value for all modification events watch on 0 should just start watch from the latest revision. recovery the initial state (thus the sync or the initial get) should be a separate operation. But, yea, I guess we need to preserve this at least for now.
- yes, I think there is general agreement for it, though we can't change that in v1 API. There is more-than-1-year old issue for that:
For this case, it won't fail. It returns empty result.


Question: } if sgNeedsUpdate { glog.V(3).Infof("delete(%s): sg(%s) - updating", serviceName, az.SecurityGroupName) <ins> // azure-sdk-for-go introduced contraint validation which breaks the updating here if we don't set these // to nil. This is a workaround until https://github.com/Azure/go-autorest/issues/112 is fixed sg.SecurityGroupPropertiesFormat.NetworkInterfaces = nil sg.SecurityGroupPropertiesFormat.Subnets = nil </ins> _, err := az.SecurityGroupsClient.CreateOrUpdate(az.ResourceGroup, *reconciledSg.Name, reconciledSg, nil) if err != nil { return err

Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Azure / Azure Container Service OS (e.g. from /etc/os-release): Xenial Kernel (e.g. ): latest lts Install tools: kubelet + static manifests Others: n/a What happened: I created a Service (type=LoadBalancer). The load balancer did not complete provisioning and security group was not updated, so the LB that is there doesn't work. What you expected to happen: Azure load balancer is created, external IP field is populated (doesn't show as "Pending") How to reproduce it (as minimally and precisely as possible): Try it. Anything else do we need to know: No. There is an issue opened against the repo that caused this problem: This was reported here: CC:
Should we do a fix that simply sets the 2 properties to nil before a real fix comes into go-autorest?
I'd prefer to fix it properly in but as of yet, it's unclear what the timeframe is. Let me ask for a timeline.
I agree that a proper fix in go-autorest would be preferable. But as long as it's not fixed, LBs are broken on k8s=1.5.2. That's why I would like to do a workaround until the real fix comes in.
Is there a workaround? Is there a way I can manually intervene to make the LoadBalancer work?
No. Not currently, other than building your own copy of with patch applied. I'm testing a more proper fix today or tomorrow - hope to have it in for 1.5.3.


Question: import ( "fmt" mathrand "math/rand" <del> "os/exec" </del> "strings" "time"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig storage What happened: Offshoot of . Let's change the tests to use the programmatic API instead of gcloud CLI.
Just confirming that this issue is referring only to the 3 gcloud CLI calls in
yup
PR:


Question: {Name: "containerA"}, {Name: "containerB"}, }, <del> } currentState := api.PodStatus{ Host: "machine", </del> <ins> RestartPolicy: api.RestartPolicy{Always: &api.RestartPolicyAlways{}}, </ins> } runningState := api.ContainerStatus{ State: api.ContainerState{

I'm actually not sure if it's by design. But certainly replication controller doesn't behave like it expects this.
If restart policy is to restart always, they should be restarted, but in that case they shouldn't be reported as failed.
This is a known issue, and by design based on upper-level will observe the status reported from kubelet, but not check with RestartPolicy and generate PodCondition / PodStatus. When RestartPolicy is Always, even Kubelet reports all containers are at failed state, apiserver shouldn't mark Pod as failed state. This is why initially I want kubelet report Pod state / status / condiction.
As a quick fix, we could let apiserver populate the right state / status based on containerStatus and RestartPolicy for now. In a long term, I still think we should make kubelet smarter and atomic and self-contained component, not just a wrapper around docker.
Short term fix should be P1 then we can de-prio for a "realer" fix.
Yes, I am working on the short term fix, and should have it included in 0.6 cut.
are you still working on this?
Yes, I am working on this. Should finish it today.


Question: t.Fatalf("Error constructing: %v", err) } <del> defProf := sched.Profiles["default-scheduler"] </del> <ins> defProf := sched.Profiles[v1.DefaultSchedulerName] </ins> gotPlugins := defProf.ListPlugins() if diff := cmp.Diff(tc.wantPlugins, gotPlugins); diff != "" { t.Errorf("unexpected plugins diff (-want, +got): %s", diff)

has compatibility tests for Policy v1. We are lacking similar tests for other options of , such as: [x] : This can be right away to test that default plugins can be disabled, changed order, etc. [x] : This one is trickier, as it requires a new method in to expose all configurations and a new method in so that they can say which args they used. /sig scheduling /help /kind cleanup
/assign
you can work on the first one. I'll have a PR soon that should clarify how the second one should be done.
I have sent a pr for the first one. And I'm working on the second one, I think it's better to keep 2 PRs for this issue? If so, PTAL at the first one, thanks.
Yes, 2 PRs is ideal. I'll review later today.
I have one small question about do I need to mock some fake plugins for testing , or just testing the default plugins?
You should use the existing plugins, because these are compatibility tests.
Okay
Actually, I think adding a new method such as to interface is a little redundant. In fact, most plugins don't need any argument for building them, so we are talking about adding a new method that needs to be implemented by all of the existing and coming plugins. And users who write out-of-tree plugins also hardly have the need for using this method. IMO, it would be more meaningful to move up to the interface. Then by this method, we can simply implement for default plugins who actually need args. WDYT?
should remain unexported because it provides pointers. However, I agree that might be too much to ask for all plugins. Instead, we can add a new interface and then plugins can opt-int. Not sure if we should follow this extension pattern Any thoughts
On the other hand, simply returning doesn't seem like much.
Sure, then I will add like what we want before(add this method to interface), and if there is any better idea later, please let me know.
I think we should just add it to the plugins that actually have args, this will require custom code in the test, but I think it is better than having all plugins implement a function means something to only 1% of them.
Thanks, but now there is a problem that we can't get the registered plugins with current API.

So, I'm not sure if we can just implement the method in with some reflect code for checking if one plugin has implement method? This could help us with keeping the interface clean. cc
you don't need to check if it implements BuildArgs, since this is for testing, we can hardcode which plugins should have their args verified.
The test will not be pretty, but I think it is worth the tradeoff.
Do you mean that we don't test the default plugins, but define some custom plugins in the test file? And we don't need the ListPluginArgs method anymore, am I right?
and I discussed this offline and arrived to this conclusion: For the compatiblity test, we can mock ( in ) to capture the arguments the configuration that is being passed. This can also be used by with We want another test for that checks the build args for the in-tree plugins accessing the internal pointers. These 2 tests should cover the whole stack. Sorry for the back-and-forth. Sometimes it's useful to see a full PR to realize that the initial idea wasn't good enough. Thanks a lot for it.
Sure, that sounds much more clear, thanks. :D
One more question, the is under package , and the compatibility test is under package . So for mocking , we need to change to a function variable in , and add a method for it, right It will look like: Maybe we can add something like to , but that's the implementation details. I want to make sure if the above design is proper.
Or rather, you could add something like: . Then you can call the capturer from if there is one defined.
Got your message in slack, let me reply here. sounds good, and do I need to revert the change in ? I think is not necessary now.
I think they will still be useful for the other test:
Hello are you still working on this?
Ah, I'm so sorry that I'm so busy in the last few weeks, but I will send prs fot this issue in this weekend. Sorry for my delay again.


Question: return a.updateStatusIfNeeded(hpaStatusOriginal, hpa) } <ins> // normalizeDesiredReplicas takes the metrics desired replicas value and normalizes it based on the appropriate conditions (i.e. < maxReplicas, > // minReplicas, etc...) func (a *HorizontalController) normalizeDesiredReplicas(hpa *autoscalingv2.HorizontalPodAutoscaler, currentReplicas int32, prenormalizedDesiredReplicas int32) int32 { var minReplicas int32 if hpa.Spec.MinReplicas != nil { minReplicas = *hpa.Spec.MinReplicas } else { minReplicas = 0 } desiredReplicas, condition, reason := convertDesiredReplicasWithRules(currentReplicas, prenormalizedDesiredReplicas, minReplicas, hpa.Spec.MaxReplicas) if desiredReplicas == prenormalizedDesiredReplicas { setCondition(hpa, autoscalingv2.ScalingLimited, v1.ConditionFalse, condition, reason) } else { setCondition(hpa, autoscalingv2.ScalingLimited, v1.ConditionTrue, condition, reason) } return desiredReplicas } // convertDesiredReplicas performs the actual normalization, without depending on `HorizontalController` or `HorizontalPodAutoscaler` func convertDesiredReplicasWithRules(currentReplicas, desiredReplicas, hpaMinReplicas, hpaMaxReplicas int32) (int32, string, string) { var minimumAllowedReplicas int32 var maximumAllowedReplicas int32 var possibleLimitingCondition string var possibleLimitingReason string if hpaMinReplicas == 0 { minimumAllowedReplicas = 1 possibleLimitingReason = "the desired replica count is zero" } else { minimumAllowedReplicas = hpaMinReplicas possibleLimitingReason = "the desired replica count is less than the minimum replica count" } // Do not upscale too much to prevent incorrect rapid increase of the number of master replicas caused by // bogus CPU usage report from heapster/kubelet (like in issue #32304). scaleUpLimit := calculateScaleUpLimit(currentReplicas) if hpaMaxReplicas > scaleUpLimit { maximumAllowedReplicas = scaleUpLimit possibleLimitingCondition = "ScaleUpLimit" possibleLimitingReason = "the desired replica count is increasing faster than the maximum scale rate" } else { maximumAllowedReplicas = hpaMaxReplicas possibleLimitingCondition = "TooManyReplicas" possibleLimitingReason = "the desired replica count is more than the maximum replica count" } if desiredReplicas < minimumAllowedReplicas { possibleLimitingCondition = "TooFewReplicas" return minimumAllowedReplicas, possibleLimitingCondition, possibleLimitingReason } else if desiredReplicas > maximumAllowedReplicas { return maximumAllowedReplicas, possibleLimitingCondition, possibleLimitingReason } return desiredReplicas, "DesiredWithinRange", "the desired count is within the acceptable range" } func calculateScaleUpLimit(currentReplicas int32) int32 { return int32(math.Max(scaleUpLimitFactor*float64(currentReplicas), scaleUpLimitMinimum)) } </ins> func (a *HorizontalController) shouldScale(hpa *autoscalingv2.HorizontalPodAutoscaler, currentReplicas, desiredReplicas int32, timestamp time.Time) bool { if desiredReplicas == currentReplicas { return false

We recently had 2 serious bugs caused by non-trivial logic calculating desiredReplicas in hpa reconcileAutoscaler method (, ). I think we should extract this part of logic (starting around ) to a separate function and make sure it's covered with unittests for different possible edge condition. The logic itself takes a few ints as input (calculated desiredReplicas, min/max replicas, currentReplicas, ...) to produce a single integer value and set some status conditions, making it very easily unittestable if we extract it to a separate function. WDYT
+100. It would be super unfortunate if we had yet another issue in this area of the code.
agreed. you've been going over these pieces of code recently, you want to tackle this?
Yes, I'd love to! Thanks for the ping. I'll tackle this this weekend :)


Question: wm[number] = w } <del> func (wm watchersMap) deleteWatcher(number int) { delete(wm, number) </del> <ins> func (wm watchersMap) deleteWatcher(number int, done func(*cacheWatcher)) { if watcher, ok := wm[number]; ok { delete(wm, number) done(watcher) } </ins> } func (wm watchersMap) terminateAll(done func(*cacheWatcher)) {

A data race manifested in job link: NA NA No response No response No response /sig api-machinery
/cc
/assign
ack, fixing ...
/triage accepted


Question: class:         makeClass(nil, topologyLackOfValues), shouldSucceed: false, }, <ins> "duplicate TopologySelectorRequirement values": { class:         makeClass(nil, topologyDupValues), shouldSucceed: false, }, "multiple TopologySelectorRequirement values": { class:         makeClass(nil, topologyMultiValues), shouldSucceed: true, }, "empty MatchLabelExpressions": { class:         makeClass(nil, topologyEmptyMatchLabelExpressions), shouldSucceed: false, }, "duplicate MatchLabelExpression keys": { class:         makeClass(nil, topologyDupKeys), shouldSucceed: false, }, "duplicate MatchLabelExpression keys but across separate terms": { class:         makeClass(nil, topologyMultiTerm), shouldSucceed: true, }, "duplicate AllowedTopologies terms - identical": { class:         makeClass(nil, topologyDupTermsIdentical), shouldSucceed: false, }, "two AllowedTopologies terms, with a pair of the same MatchLabelExpressions and a pair of different ones": { class:         makeClass(nil, topologyExprsOneSameOneDiff), shouldSucceed: true, }, "two AllowedTopologies terms, with a pair of the same Values and a pair of different ones": { class:         makeClass(nil, topologyValuesOneSameOneDiff), shouldSucceed: true, }, "duplicate AllowedTopologies terms - different MatchLabelExpressions order": { class:         makeClass(nil, topologyDupTermsDiffExprOrder), shouldSucceed: false, }, "duplicate AllowedTopologies terms - different TopologySelectorRequirement values order": { class:         makeClass(nil, topologyDupTermsDiffValueOrder), shouldSucceed: false, }, } // Disable VolumeScheduling so nil VolumeBindingMode doesn't fail to validate. err := utilfeature.DefaultFeatureGate.Set("VolumeScheduling=false") if err != nil { t.Fatalf("Failed to disable feature gate for VolumeScheduling: %v", err) </ins> } // TODO: remove when feature gate not required <del> err := utilfeature.DefaultFeatureGate.Set("DynamicProvisioningScheduling=true") </del> <ins> err = utilfeature.DefaultFeatureGate.Set("DynamicProvisioningScheduling=true") </ins> if err != nil { t.Fatalf("Failed to enable feature gate for DynamicProvisioningScheduling: %v", err) }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --/kind feature The field essentially has the type , but currently there is no validation for set and map semantics, i.e. unique elements / keys. Having this validation at API level removes logic to validate at the volume plugin level. edit: this check will not be in cases where this makes it difficult to add other fields and make this field optional. /sig api-machinery /sig storage
/assign
Thanks for catching this. Would you be able to implement this for this release? We want to try to go beta if we can, and we won't be able to strengthen validation once that happens.
Yes will do!


Question: return cmdutil.AddSourceToErr("creating", info.Source, err) } info.Refresh(obj, true) <del> metadata, err := meta.Accessor(info.Object) if err != nil { return err } visitedUids.Insert(string(metadata.GetUID())) </del> } <ins> metadata, err := meta.Accessor(info.Object) if err != nil { return err } visitedUids.Insert(string(metadata.GetUID())) </ins> count++ if printObject {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The dry run output listed all the resources that would be configured: deployments, HPAs, services, and an ingress: This was expected. However, it then listed all resources again (except for the HPAs) as being pruned: The config was then applied with: As expected, no resources were actually pruned. What you expected to happen: should actually display which resources are going to be pruned. How to reproduce it (as minimally and precisely as possible): a YAML file to with it: run + prune: should be listed as both configured and pruned it again with pruning: that the deployment is configured but not pruned, even though the dry run said it would be. Anything else we need to know?: Was trying to write a devops script that gave a warning before any resources would be pruned. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): (Lubuntu 18.04) Kernel (e.g. ):Install tools: Others:
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I can reproduce it, and i will try to fix it.


Question: type VolumeOptions struct { CapacityGB int <del> Tags       *map[string]string </del> <ins> Tags       map[string]string </ins> } // Volumes is an interface for managing cloud-provisioned volumes

We should tag EBS volumes created by k8s with our tags, just like we (should) tag everything else we create/own.
Is there any documentation yet on automatic EBS volume creation?
from a user perspective, they are not AWS specific. That said, I don't know that we have clear user-facing docs. Here's an example though, from my (very) WIP Ubernetes-Lite docs: ... The PV is also tagged with the zone & region is is in (us-central1-a): Create a pod that uses the PVC:
It doesn't seem to cleanup the volume when the PV is deleted... probably intended behavior. Any way to tell it to automatically delete the volume on PV delete?
- I'm not sure about foo - hopefully that'll be resolved with docs. I think the failure to delete the volume is actually pretty important. Do you want to open an issue? It's not AWS specific (and this particular issue was auto-closed when the PR merged)
yeah I will open an issue later today. Issue seems to still be happening - the PV is never deleted. On Sat, Mar 12, 2016, 6:24 AM Justin Santa Barbara wrote:


Question: "github.com/spf13/cobra" <ins> utilflag "k8s.io/apiserver/pkg/util/flag" </ins> kubeadmapi "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm" kubeadmapiext "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1alpha1" "k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/validation"

/kind feature What happened: When calling , there is no way to override default flag values for the different components of the controlplane. This is surprising, as in the config structure (and in controlplane phase codebase itself), there are explicit settings for doing that (APIServerExtraArgs, ControllerManagerExtraArgs, SchedulerExtraArgs). What you expected to happen: Those config fields should be exposed as flags on and How to reproduce it (as minimally and precisely as possible): Try to deploy K8s using Kubeadm with a custom Admission Control setting (I'll follow up on this issue with a PR)
btw, I am not sure which sig owns kubeadm.
it's sig-cluster-lifecycle.
/sig sig-cluster-lifecycle
/sig cluster-lifecycle


Question: closeChan   chan bool dataStream  *fakeStream errorStream *fakeStream <ins> streamCount int </ins> } func newFakeConnection() *fakeConnection {

The port forwarding client leaks memory when a port forwarding session is reused over time. The port forwarder client should not leak memory. forwarding a port with kubectl. and close many connections to the forwarded port. The amount of memory leaked for each connection is quite small, so many connections are needed to observe the leak. This is related to , which was reported for the kubelet and fixed in . However, the same problem happens to the client. The client needs to call on the , otherwise the streams created by will be kept in memory until the port forwarder itself is closed, even if the streams themselves closed long before. <details</details<detailsN/A </details<details</details<details</details<details</details<details</details
/sig api-machinery
/cc /triage accepted
/cc


Question: csiNodeResource   = storageapi.Resource("csinodes") ) <ins> func (r *NodeAuthorizer) RulesFor(user user.Info, namespace string) ([]authorizer.ResourceRuleInfo, []authorizer.NonResourceRuleInfo, bool, error) { if _, isNode := r.identifier.NodeIdentity(user); isNode { // indicate nodes do not have fully enumerated permissions return nil, nil, true, fmt.Errorf("node authorizer does not support user rule resolution") } return nil, nil, false, nil } </ins> func (r *NodeAuthorizer) Authorize(ctx context.Context, attrs authorizer.Attributes) (authorizer.Decision, string, error) { nodeName, isNode := r.identifier.NodeIdentity(attrs.GetUser()) if !isNode {

What happened: I checked if kubelet () has permissions for with SelfsubjectAccessReviews and I saw that it has. But when I checked its permissions with SelfsubjectRulesReviews it didn't show it. What you expected to happen: I expected that the API SelfsubjectRulesReviews will show that has permssions for . How to reproduce it (as minimally and precisely as possible): You can check it on the master or the worker. Test SelfsubjectAccessReviews: You will get answer that it has permissions: Test SelfsubjectRulesReviews: It won't show the permissions. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm /area/api /kind/bug /sig api-machinery
/sig api-machinery
This is a more clean display that show the bug: I think the problem is related to the fact that the file contain the kubelet certificate which is part of the group :
/sig auth
What is the SelfSubjectRulesReview response from the server ()? Note that not all authorizers support enumerating permissions, and the API object status includes an attribute which explicitly indicates the result is incomplete. See
It has Details:
I noticed that it also doesn't show that it has permissions to view and . It probably related to the Node authorizer: Because I assume that these rules doesn't appear in RBAC.
The node authorizer does not enumerate permissions, so it does not contribute to SelfSubjectRulesReview results
fixes setting correctly when the API is requested with a kubelet's credentials


Question: postTimeoutFn := func() { cancel() <del> metrics.Record(req, requestInfo, metrics.APIServerComponent, "", http.StatusGatewayTimeout, 0, 0) </del> <ins> metrics.RecordRequestTermination(req, requestInfo, metrics.APIServerComponent, http.StatusGatewayTimeout) </ins> } return req, time.After(timeout), postTimeoutFn, apierrors.NewTimeoutError(fmt.Sprintf("request did not complete within %s", timeout), 0) }

What happened: When the kube-apiserver timeouts on a request, we end up doubling the counts measured in : apiserverrequestcounts apiserverrequesttotal apiserverrequestlatencies apiserverrequestlatenciessummary apiserverrequestdurationseconds apiserverresponsesizes What you expected to happen: We only count our requests a single time in our metrics. How to reproduce it (as minimally and precisely as possible): Add a 60 second sleep in the func . You may want to filter by resource (like pods or something) and verb for testing (otherwise this will be quite annoying). Manually hit that endpoint and look at the metric output. /assign
/sig api-machinery instrumentation /cc
Thanks for filing. Can we, while fixing this, add a metric counting timed-out requests? You can't tell from the latency distribution, even if we made the buckets more useful.
Yeah, that's the current plan. I've relayed the idea to offline and he has concurred.


Question: glog.Errorf("Could not mark device is mounted to actual state of world: %v", err) continue } <ins> glog.Infof("Volume: %v is mounted", volume.volumeName) </ins> } _, err = rc.desiredStateOfWorld.AddPodToVolume(volume.podName, volume.pod, volume.volumeSpec,

/kind bug What happened: Volumes used by pod are not getting mounted after rebooting a node. Effects Flexvolume/FC/ISCSI volume types. Pod goes into running state using local file system path. What you expected to happen Volume should be mounted and pod should not go into running state if the volume fails to attach. How to reproduce it (as minimally and precisely as possible): Create a RS with Flexvolume/FC/ISCSI. Reboot the node. Volume would not be mounted after reboot. Anything else we need to know?: Kubelet on a process restart avoids remounting the volume. The same code path is used on a node reboot and it avoids remounting the volume. GCE, AWS etc seems to be working by chance as the SPEC reconstruction fails when a volume is not mounted on reboot. Environment: Kubernetes version (use ): 1.6.9 Cloud provider or hardware configuration**: Baremetal OS (e.g. from /etc/os-release): Centos 7.2 Kernel (e.g. ): 3.10.0-327.el7.x86_64
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig storage
/cc


Question: t.Errorf("diff: %s", util.ObjectDiff(e, a)) } expect := fields.Set{ <ins> "metadata.name":                  "f0118", </ins> "involvedObject.kind":            "Pod", "involvedObject.name":            "foo", "involvedObject.namespace":       "baz",

Using a watch expression on endpoints can lead to unpredictable behavior that includes returning a watch stream that ignores filters (e.g. a {name} determination) and returns items of the wrong type (e.g. changes to PODS in the ENDPOINTS stream). returns This result includes updates for services "kubernetes", "kubernetes-ro", "redis-slave", "redis-master", etc. Even though the requesting URL indicated only "guestbook" updates were to be watched. This seems to only occur when no resourceVersion is specified and the most recent resourceVersion has fallen off the etcd watch window. The expected behavior when resourceVersion is not specified I would think should be either: an initial update with the latest version of the resource, followed by filtered updates. nothing, followed by filtered updates. Though I'm not sure which of these is actually the spec. I would prefer (1) since it allows you to do a resource watch without having to first do a separate GET. But (2) coupled with a preceding GET would yield the same results.
IRC: cc
The pods mentioned in the stream are references to the pods corresponding to endpoints. See for discussion behind the feature. We probably need to distinguish the resourceVersion at which an object was last modified, which is also used as an update precondition, vs. the current (global) resourceVersion needed for watch.
Does the change to use GET with watch as a query parameter enable the GET to both return the latest version and subscribe to changes, or is a separate non-watch GET still required?
/subscribe
If you pass resourceVersion=0 you get list then watch (for single or multiple).
That behavior has been there for a while - the only challenge is you don't know when the list is complete, and we felt that list then watch more clearly delineated that.
This.
Watching with resourceVersion=0 should return an error, IMO.
I think the semantics for {watch a single resource} vs. {watch all resources of the same type} should be separated. When watching a single resource I would expect the watch to have consistent semantics. That is, I would expect it to be possible to watch an item and be guaranteed either: I saw every change to the item, or I knew that a change was missed (and so could do a raw GET to re-sync).

To this end it would be ideal if watch for a single item had the following semantics: Delivered the current state, followed by a list of all changes. If any change is missed or can't be delivered then the watch is terminated. Delivers either a guarantee of all changes that occurred on that resource from (exclusive) to present. If the server cannot prove that there were no missing changes then it should return an error and terminate the watch. This would imply that: Would always return an error unless the server can prove that it can deliver all changes to the resource (e.g. when the resource hasn't been created yet.) I'm surprised that the interface wants to provide multi-resource watch. It would seem that the semantics of this watch would either be potentially expensive to deliver or "best effort" such that it is basically useless for anything other than casual human consumption. Imagine a future where the master is sharded such that even different resources of the same type are on different shards. To provide meaningful semantics to a multi-resource watch would require that resource versions be meaningful in a global sense and in a way that spans all shards. Wouldn't this be undesirable? It would require that any write at any shard would need a round-trip message to all other shards to ensure that the logical clocks remain in sync. A multi-watch would similarly have to be fanned out to all shards in the backend. Both of these would defeat the benefits of sharding the data in the first place. I could understand providing a multi-resource watch for the creation and deletion of resources of a particular type (handled potentially by a secondary index in the sharded scheme) without providing a full watch of the contents. You could then use this watch to establish resource-specific watches for the resources you needed to track the contents of. Maintaining a secondary index wouldn't have to be transactional with changes that produced or destroyed items.
resourceVersion=0 is the same thing as saying watch resourceVersion=ObjectModifiedResourceVersion-1. Perhaps it should be more explicit, but it is generally useful. Why is blind watch bad? ----- Original Message -----
Also, I don't see it mentioned above but - everything in  is ignored. Confusing, but already fixed by the deprecation of .

cc
We don't offer multi-resource watch today, but in the future you're likely to correlate very strongly with a shard based on your namespace, rather than a shard on your type. I don't think sharding by type will be common except on a few types of resources (events, namespaces, possibly endpoints). The access pattern for most clients is going to be "in namespace Foo, get type X, get type Y, get type Z, get type A". Most UIs that want to watch anything will watch on the namespace. So the access pattern is less likely to reward sharding by type.
The query: Is a multi-resource watch. It watches for all resources of type Endpoints. But do you want the interface to guarantee that all objects in the same namespace are updated with consistency? I agree that shard boundaries are almost certainly NOT type-boundaries. For instance there is clearly a consistency relationship between a Service and its Endpoints object. I would hope that it would be impossible for me to observe a change to a Service object and then to subsequently query for its Endpoints object and get a stale read (semantically speaking). But that is a query across types. Suggesting that the appropriate shard boundaries are certainly not type-boundaries.
Sorry, I was referring to "multi-resource" as "watching over multiple resource types". It is possible to emulate a watch across a small number of stable shards by creating a resource version tuple (a vector clock, essentially) - for medium sized clusters I see that as a reasonable option. It's more likely though that infrastructure components that want to "watch the world" are either themselves shardable (such that it's acceptable for world to be defined as "my shard") or an intermediate component supports them. That intermediate component might be something like a partially ordered event stream that provides a firehose to the cluster as an aggregator of the cluster changes. Today, we require that API consumers assume resource version is internally consistent across a single type in a single namespace. Controllers are allowed to watch the world until such a time as we implement a sharding solution. There is no consistency relationship across resource types - services and endpoints are independent objects with independent lifecycles. It is by design that Endpoints can be updated out of band with Services, to allow downstream components to offload them.

We relax referential rules across resource types - there is no guarantee a referenced object will be created before or after you, or that it should exist only if you exist.
I completely agree there are possible implementations that can tolerate sharded and/or inconsistent queries. And deciding that certain kinds of JOINs, particularly across resource types and/or namespaces, are always going to be inconsistent seems like a fine place to land. I was merely surprised that the API was designed to promote the appearance of consistency for these queries today. I would have expected that the API would be more conservative to reserve these degrees of freedom around consistency for future implementations without requiring a future breaking change. Offering less functionality in V1 or functionality with weaker semantics would allow weakly consistent future implementations to emerge later without breaking existing applications. As a general rule I try to avoid APIs that give the appearance of consistency if they are not intended to actually deliver consistency. I think most developers will always assume consistency unless there is a really good reason to believe otherwise. This seems like human nature since people expect systems to exhibit the same causal behavior as the real world. Take for instance Endpoints. My impression of this after seeing the API was that Endpoints was simply a VIEW (in the SQL since) that provides a convenient way to express the JOIN between Service and Pods. From that perspective I would certainly have expected there to be a causal relationship between a change in the Service and its corresponding Endpoints object. I could easily imagine myself making that mistake when writing a application. Perhaps my program changes the Service object and then wants to send a message to all of the Endpoints associated with the new Service definition. After changing the Service, my program queries Endpoints. Did it get the new set of Endpoints? There are several possible outcomes: The Endpoints is the same as it was before I changed the Service Definition: The new Service definition just happens to result in the same set of Endpoints as before. I have not waited long enough for Endpoints to be updated. How long should I wait? What indicates that I've waited long enough? The Endpoints is different: Is this the correct one? The one that corresponds to my Service definition change? Or is this the update that occurred due to a different Service definition change that happened just before mine? If so, I should wait longer for mine.

Or is this the update that occurred due to a different Service definition change that happened just after mine? If so, I should either use this one or attempt my Service definition change again. In an inconsistent and asynchronously updated world we need to provide a way to answer these questions. (e.g. perhaps Endpoints should include the the resourceVersion of the Service definition used to compute it.) All applications need to properly deal with this extra information. (e.g. need to ignore all updated Endpoints until one arrives that has a resourceVersion = their change to the Service definition.) I'd imagine it's possible to implement this and for customers to use it correctly, but I think it offers a lot of opportunity for bugs. Users might simply wait for the next update and then take the wrong action leading to a misbehaving system. If we could decide which relationships should exhibit consistency and provide it where the developer would expect it then the API is simply and easier to use and applications are more likely to be correct. If we don't offer APIs which might lead the developer to make incorrect assumptions then applications are less likely to have those bugs.
I dug into this... it's more broken than I thought it was, so I've made a fix in . Once my fix goes in, a single object acts as if you didn't pass a name: this will be fixed directly (at least for the resource types I checked). get an appropriate resource version, you will have to LIST with a fieldselector ("?fields=name%3Dfoo" or some such-- don't recall exact syntax) instead of GETting. This way we don't have to modify the API to pass a special "watch at" resource version.


Question: stripped := []byte{} lines := bytes.Split(file, []byte("n")) for i, line := range lines { <del> if bytes.HasPrefix(bytes.TrimSpace(line), []byte("#")) { </del> <ins> trimline := bytes.TrimSpace(line) if bytes.HasPrefix(trimline, []byte("#")) && !bytes.HasPrefix(trimline, []byte("#!")) { </ins> continue } stripped = append(stripped, line...)

configmap, like edit configmap test, and make some error and write-save, will see below when the content after edited is invalid, should not remove data , thus mean expect content below configmap 2 edit configmap make invalid No response version kubeadm <details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
PTAL
Hi, I am new to this project and would love to contribute. Can I work on this issue?
/sig cli


Question: apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" <ins> authauthenticator "k8s.io/apiserver/pkg/authentication/authenticator" </ins> "k8s.io/apiserver/pkg/authentication/group" "k8s.io/apiserver/pkg/authentication/request/bearertoken" <ins> authenticatorunion "k8s.io/apiserver/pkg/authentication/request/union" </ins> "k8s.io/apiserver/pkg/authentication/user" "k8s.io/apiserver/pkg/authorization/authorizer" "k8s.io/apiserver/pkg/authorization/authorizerfactory"

/kind failing-test <!-- If this is a flake, please add: /kind flake --/kind flake <!-- Please assign a SIG using: /sig SIG-NAME --/sig api-machinery
/assign I want to take a try
2 but actual 0 till timeout resource services, expected 2, actual 0 error waiting or ResourceQuota status: timed out waiting for the condition time later, actual state became 2 Seems polls a ResourceQuota status with timeout 10s may be not enough, we should enlarge it into 300s, that is poll for 300s timeout instead of 10s.
/triage accepted