Question: imageutils "k8s.io/kubernetes/test/utils/image" ) <ins> // secondNodePortSvcName is the name of the secondary node port service const secondNodePortSvcName = "second-node-port-service" </ins> var ( // agnHostImage is the image URI of AgnHost agnHostImage = imageutils.GetE2EImage(imageutils.Agnhost)

What happened: indicates that we're missing an e2e test for. This specific issue is just a tracker to create a new E2E test that demonstrated .
/sig network
/assign
btw, should this be a part of conformance?
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I can start working on this.
Sure/ are you working full time on it ? If so I'll drop the issue from my work ... Otherwise we can collaborate as I already have some code for this. Let me know. Was hoping to have this done in the next week or so.
I would love to collaborate, especially if you already have some code. But if you want to drop this from your work, I don't mind taking it on full time either. I can pick up from where ever you left off.
ok, if youve got full bandwidth go for it. if make progress on it this wk ill just drop it . I actually dont know where my old branch is for this :), so i guess youll have to start from scratch. One thing to note ~ Might be seeing if any tests can be reused for this or extended in a simple way by just adding a secondary service, rather then writing a full blown test from scratch. Let me know how it goes thanks !
I will let you know how it progresses. Thanks !
thanks for getting this taken care of :) looks like the PR is LGTM.
Thanks for letting me take a stab at it!


Question: endpointSlice.ObjectMeta.Name = fmt.Sprintf("%s-%s", endpointSlice.ObjectMeta.GenerateName, rand.String(8)) endpointSlice.ObjectMeta.GenerateName = "" } <ins> endpointSlice.ObjectMeta.ResourceVersion = "100" </ins> return false, endpointSlice, nil })) <ins> client.PrependReactor("update", "endpointslices", k8stesting.ReactionFunc(func(action k8stesting.Action) (bool, runtime.Object, error) { endpointSlice := action.(k8stesting.CreateAction).GetObject().(*discovery.EndpointSlice) endpointSlice.ObjectMeta.ResourceVersion = "200" return false, endpointSlice, nil })) </ins> return client }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: With EndpointSlice feature enabled, after EndpointSliceController creates or updates an EndpointSlice when reconciling a Service, the Service would always be processed one extra time, which is triggered by the EndpointSlice /updated event made by itself during the first processing. What you expected to happen: EndpointSliceController shouldn't reprocess a Service due to the EndpointSlice update made by itself. How to reproduce it (as minimally and precisely as possible): EndpointSlice feature, improve kube-controller-manager log to 5. a nginx deployment a Service backed by the above deployment. kube-controller-manager log: the service will be processed twice in succession. Anything else we need to know?: It's because EndpointSliceController tracks stale resource version after creating/updating an EndpointSlice. Environment: Kubernetes version (use ): v1.18.0-alpha.3 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/assign
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: if !ok { return 0, fmt.Errorf("invalid max log size") } <del> if maxSize < 0 { return 0, fmt.Errorf("negative max log size %d", maxSize) } </del> return maxSize, nil }

Setting a negative value in currently makes kubelet crash with a stack trace. The function raises an error when the value is negative, but the comment on  says that setting a negative value disables log rotation. At the moment there’s no way to disable log rotation and I believe removing lines 141 to 143 from , would allow this.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node (I guess)
/area kubelet can you show the stack trace?
The stack trace is 5000 lines long and apart from the following message, I don’t think it is relevant:
/assign
You can try to quote the value with double quotation when the value is negative, like the following: syncFrequency: 0s volumeStatsAggPeriod: 0s containerLogMaxSize: "-1"
I tried "-1", but still error
volumeStatsAggPeriod: -1s means disable
Which Kubernetes version have you installed? As if on my v1.19.4 system,no errors occured after I quoted the value with double quotation.
v1.20.2
I’m on v1.20.4. That’s already the case. Also, can only be set as a string in kubelet’s configuration, trying to set an integer will raise a parsing error (see ). The problem is not with the configuration, I’ve linked to the exact line in the code that checks for a negative value and raises the error.
Maybe you can try to disable this featureGate
Seems to work, I don’t know why I didn’t find this feature gate before.


Question: gvr("flowcontrol.apiserver.k8s.io", "v1beta3", "prioritylevelconfigurations"): { Stub:             `{"metadata": {"name": "conf4"}, "spec": {"type": "Limited", "limited": {"nominalConcurrencyShares":3, "limitResponse": {"type": "Reject"}}}}`, ExpectedEtcdPath: "/registry/prioritylevelconfigurations/conf4", <del> ExpectedGVK:      gvkP("flowcontrol.apiserver.k8s.io", "v1beta2", "PriorityLevelConfiguration"), </del> }, // --

adds new version for in 1.26. For 1.26, we are setting the storage version to because vN-1 understands that level. In 1.27, we should change the storage version from to .
/sig api-machinery
/cc
/cc
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:


Question: ## Salt cluster setup <del> The **salt-master** service runs on the kubernetes-master node. </del> <ins> The **salt-master** service runs on the kubernetes-master node [(except on the default GCE setup)](#standalone-salt-configuration-on-gce). </ins> The **salt-minion** service runs on the kubernetes-master node and each kubernetes-minion node in the cluster. <del> Each salt-minion service is configured to interact with the **salt-master** service hosted on the kubernetes-master via the **master.conf** file. </del> <ins> Each salt-minion service is configured to interact with the **salt-master** service hosted on the kubernetes-master via the **master.conf** file [(except on GCE)](#standalone-salt-configuration-on-gce). </ins> ``` [root@kubernetes-master] $ cat /etc/salt/minion.d/master.conf

We need to document all the configuration derived by Salt. Ideally this would be a markdown doc that we kept as live documentation within the project. The main reasons to bother: It's hard right now for anyone approaching the directory to get a big picture view of what's is going on, especially across providers. If we ever want to do something disciplined w.r.t. upgrade between (non-dev) versions, we're probably going to need a place to look for the state of the world. Decreasing the velocity with which people can add to Salt configuration will encourage people to delete more Salt configuration. cc


Question: // start new KMS Plugin _ = mock.NewBase64Plugin(t, "@new-kms-provider.sock") // update encryption config <del> if err := os.WriteFile(filepath.Join(test.configDir, encryptionConfigFileName), []byte(encryptionConfigWithNewProvider), 0644); err != nil { t.Fatalf("failed to update encryption config, err: %v", err) } </del> <ins> updateFile(t, test.configDir, encryptionConfigFileName, []byte(encryptionConfigWithNewProvider)) </ins> wantPrefixForSecrets := "k8s:enc:kms:v1:new-kms-provider-for-secrets:" // implementing this brute force approach instead of fancy channel notification to avoid test specific code in prod. // wait for config to be observed <del> verifyIfKMSTransformersSwapped(t, wantPrefixForSecrets, "", test) </del> <ins> verifyIfKMSTransformersSwapped(t, wantPrefixForSecrets, test) </ins> // run storage migration // get secrets

transformation expandless | 14m1s Failed; === RUN TestEncryptionConfigHotReload TestEncryptionConfigHotReload When I see this flake, 10-09 is the first one.(But the job history in the testgrid only started from 10-03) some PR that may be related: /sig auth /cc
/cc
/triage accepted
/cc failed two times in recent 3 runs:
/reopen as of this is still .
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: NodeName             string               `json:"nodeName"` AuthorizationModes   []string             `json:"authorizationModes,omitempty"` <ins> // Mark the controller and api server pods as privileged as some cloud // controllers like openstack need escalated privileges under some conditions // example - loading a config drive to fetch node information PrivilegedPods bool `json:"privilegedPods"` </ins> Token    string           `json:"token"` TokenTTL *metav1.Duration `json:"tokenTTL,omitempty"`

Dear All: I'm not sure if I'm right. When i use whith .I found controller-manager logs Someone can help me? Thanks a lot.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Please give more information about your set up cc
Sorry for report later. My openstack didn't support metadata service,but support config-drive service. More info:
Thanks Based on this, I think it's an openstack cloudprovider problem more than a kubeadm problem...
Thank you for help me. I will change the label.
/area cloudprovider
/remove-kind support
Should be fixed with you can specify a search order for metadata and config drive /close
Specifying a search order doesn't really solve the problem, especially if the metadata api isn't available which is what ZhenyangZhao said . I'm getting the same issue and since the config drive is my only option at the moment.
Like for this to be /open
can you please provide logs/versions for your problem in your environment? Thanks, Dims
Let me know if you need anything else Kubernetes version: 1.8.4 Docker version: 17.09.0-ce Linux: CentOS 7.4 Looking at the docker container log for the api-server (I don't have problems with the controller-manager):
and share the following:
Hi! Thank you for filing this issue. We appreciate your interest in Kubernetes and kubeadm. In order to make the kubernetes/kubernetes issue tracker more usable, we're routing issues to different repositories with different focuses. I think this issue belongs in the kubeadm repository. Please move this issue to the in order for the kubeadm team to be able to track this efficiently. If you disagree with my assessment of moving this issue, please speak up and/or reopen the issue. Finally, please try to reproduce the bug with the latest Kubernetes version, it might as well already have been fixed :) Thank you and have a good day!


Question: server, requestGetter := newHTTP2ServerWithClient(handler, requestTimeout*2) defer server.Close() <del> // we send two requests, each with a client timeout of requestTimeout*2 seconds // this ensures the test does not block indefinitely if the server does not respond. //  - first request (expected to timeout as designed) sent from a new goroutine //  - second request (expected to be enqueued) is sent from a new goroutine </del> <ins> // This test involves two requests sent to the same priority level, which has 1 queue and // a concurrency limit of 1.  The handler chain include the timeout filter. // Each request is sent from a separate goroutine, with a client-side timeout that is // double the timeout filter's limit. // The first request should get dispatched immediately; execution (a) starts with closing // the channel that triggers the second client goroutine to send its request and then (b) // waits for both client goroutines to have gotten a response (expected to be timeouts). // The second request sits in the queue until the timeout filter does its thing, which // it does concurrently to both requests.  For the first request this should make the client // get a timeout response without directly affecting execution.  For the second request, the // fact that the timeout filter closes the request's Context.Done() causes the request to be // promptly ejected from its queue.  The goroutine doing the APF handling writes an HTTP // response message with status 429. // The timeout handler invokes its inner handler in one goroutine while reacting to the // passage of time in its original goroutine.  That reaction to a time out consists of either // (a) writing an HTTP response message with status 504 to indicate the timeout or (b) doing an // HTTP/2 stream close; the latter is done if either the connection has been "hijacked" or some // other goroutine (e.g., the one running the inner handler) has started to write a response. // In the scenario tested here, there is thus a race between two goroutines to respond to // the second request and any of their responses is allowed by the test. </ins> firstReqResultCh, secondReqResultCh := make(chan result, 1), make(chan result, 1) go func() { defer close(firstReqRoundTripDoneCh) <del> t.Logf("Sending request: %q", firstRequestTimesOutPath) </del> <ins> t.Logf("At %s, Sending request: %q", time.Now().Format(timeFmt), firstRequestTimesOutPath) </ins> resp, err := requestGetter(firstRequestTimesOutPath) <del> t.Logf("RoundTrip of request: %q has completed", firstRequestTimesOutPath) </del> <ins> t.Logf("At %s, RoundTrip of request: %q has completed", time.Now().Format(timeFmt), firstRequestTimesOutPath) </ins> firstReqResultCh <- result{err: err, response: resp} }() go func() {

pull-kubernetes-unit TestPriorityAndFairnessWithPanicRecoveryAndTimeoutFilter 7/14 No response /sig api-machinery
/priority important-soon
/cc /triage accepted
and can this be related to ?
I see no direct relationship, but hey, anything's possible. We learned that with the misery in integration test failures when we merged not-really-guilty other stuff.
The expanded logging turned up some new evidence. See , where the complaint is now
The problem here is not APF, it is something in the timeout filter or the test server.
is something in APF triggering a timeout?
The flake definitely came in with on the prior commit on master: on the merge commit of
/assign
Note that the flaking test is deliberately suffering server-side timeouts. The symptom in flakes is that the test is checking that the client gets kube's designed response to a timeout while handling, a particular form HTTP response message, but sometimes the client gets something else (an http2 internal error?). is a change in metrics, and should have nothing to do with how timeouts while handling and detected or handled. I will see what I can see...
, regarding your comment above (): note that the output shown (the panic "as designed") is normal; if there are failures involved, the failing part was elided.
Also, the failures seem to be coming from one particular subtest () and that one does not involve designed panics.
On closer examination of the timeout filter, it appears that this "INTERNAL ERROR" is something that can happen normally in the course of handling a timeout. So I will adjust the test to accept this.
I have updated to include the fix for this flake.


Question: }, { "ImportPath": "github.com/google/cadvisor/api", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/cache/memory", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/client/v2", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/collector", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/common", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/docker", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/libcontainer", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/raw", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/rkt", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/container/systemd", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/devicemapper", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/events", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/fs", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/healthz", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/http", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/http/mux", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v1", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/info/v2", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/machine", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher/raw", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/manager/watcher/rkt", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/metrics", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/pages/static", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/storage", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/summary", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cloudinfo", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cpuload", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/cpuload/netlink", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/docker", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/oomparser", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/sysfs", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/sysinfo", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/utils/tail", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/validate", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/version", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/cadvisor/zfs", <del> "Comment": "v0.26.0-2-gb971fd1", "Rev": "b971fd1850f59f3e6e67842789217d9b006d6440" </del> <ins> "Comment": "v0.26.1", "Rev": "d19cc94b760cd8f150a0a5d95b404dec39a121a1" </ins> }, { "ImportPath": "github.com/google/certificate-transparency/go",

Is this a BUG REPORT or FEATURE REQUEST? (choose one): bug report Kubernetes version (use ): Environment: Cloud provider or hardware configuration: vsphere OS (e.g. from /etc/os-release):Kernel (e.g. ): 4.9.24-coreos Install tools: Others: What happened: After bringing a cluster up, there are no container metrics in the kubelet at /metrics What you expected to happen: I expect to see the metrics, but there are none How to reproduce it (as minimally and precisely as possible): Anything else we need to know: If I try and access the metrics in cAdvisor directly on port 4194 instead of on port then I see the following type of error for all of the metrics:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
This appears to be a v1.7 issue. If I run 1.6.6 I get these metrics. Seems like something has happened in 1.7. I'm running the 1.7 beta because I need nodePort for cni, so now I'm stuck.
Can you triage the issue since?
cc/
Can you help verifying is this is a new regression introduced in 1.7 cadvisor through node-e2e? Thanks!
I've tried this on both cos and cvm in 1.7 cluster, and no container metrics show up. However, I created a 1.6.4 GKE cluster, and container metrics are there. This is a regression in 1.7. Will figure out what is wrong here.
Here is a corresponding cadvisor issue
Even the user still can get the container metrics / stats from kubelet /summary api (recommended way), but since this is a regression introduced in 1.7 release, and we aren't dropping prometheus support, let's fix this for 1.7. Thanks!
Fixed in Will close this issue after building new cadvisor release and cherrypick into 1.7
Thanks for the quick find everyone! I'll be happy to give this a go on the next release!

I'm still facing this same issue. My kubernetes cluster is deployed with kubeadm on Centos 7. I run a k8S v1.8.1 cluster. When I try to fetch containers metrics this way: http://localhost:8001/api/v1/proxy/nodes/node1:0/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics http://localhost:8001/api/v1/proxy/nodes/node1:/metrics I find no metrics. kubelet is started with . If I start kubelet with . metrics are available through .


Question: fi if [[ "${OS_DISTRIBUTION}" == "gci" ]]; then <del> # If the master or node image is not set, we use the latest GCI dev image. # Otherwise, we respect whatever set by the user. </del> <ins> # If the master image is not set, we use the latest GCI dev image. # Otherwise, we respect whatever is set by the user. </ins> gci_images=( $(gcloud compute images list --project google-containers  --regexp='gci-dev.*' --format='value(name)') ) <del> if [[ -z "${MASTER_IMAGE:-}" ]]; then MASTER_IMAGE="${gci_images[0]}" MASTER_IMAGE_PROJECT="google-containers" fi if [[ -z "${NODE_IMAGE:-}" ]]; then NODE_IMAGE="${gci_images[0]}" NODE_IMAGE_PROJECT="google-containers" fi </del> <ins> MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-"${gci_images[0]}"} MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-google-containers} # The default node image when using GCI is still the Debian based ContainerVM # until GCI gets validated for node usage. NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${CVM_VERSION}} NODE_IMAGE_PROJECT=${KUBE_GCE_NODE_PROJECT:-google-containers} </ins> fi # Verfiy cluster autoscaler configuration.

As of (specifically ), running against GCE without setting a node image defaults it to containervm, whereas before it defaulted it to the master image. For example, my config of: brings up a coreos master and then launches nodes with images. Before the above-referenced change this gave me my expected coreos nodes. Was this change intentional? The easiest fix that comes to mind for me is only doing the new containervm defaulting if is gci. cc
Yeah, this was something I was concerned too while making the change. However, the decision seems to have been that we decided to default to GCI, yet still default to the debian based containervm. Also, if we are to seriously start supporting hybrid clusters, i.e., master and nodes run on different OS distros, which seems true according to , this old assumption of defaults to needs to disappear anyways. cc/ Thoughts?
That sounds reasonable to me. We clearly didn't think about your use case whilst making this change.
I don't see the issue with defaulting to non-hybrid while supporting hybrid.. Defaulting seems totally reasonable even in a world where hybrid support is possible. A user could explicitly control that they want hybrid via setting them to different values regardless of the defaulting, or wherever hybrid is toggled on different defaulting could be done. I left a few questions on the referenced original issue which, if/when answered, might help me in making a PR to fix this for my usecase more cleanly.
Agreed. I am making a fix to this.


Question: "Currently only applies to long-running requests.") fs.IntVar(&s.MasterCount, "apiserver-count", s.MasterCount, <del> "The number of apiservers running in the cluster, must be a positive number.") </del> <ins> "The number of apiservers running in the cluster, must be a positive number. (In use when --endpoint-reconciler-type=master-count is enabled.)") </ins> fs.StringVar(&s.EndpointReconcilerType, "endpoint-reconciler-type", string(s.EndpointReconcilerType), "Use an endpoint reconciler ("+strings.Join(reconcilers.AllTypes.Names(), ", ")+")")

ref: and An e2e test should be , and the default should be updated. /assign cc
/assign
Thanks I can take this on.
Could someone point me to where the manifests are for the e2e test suites, I am happy to submit the patches?
Are there API server e2e tests currently?
there are not HA API server e2e tests that I'm aware of
I opened a PR for the default changes. Are there suggestions on a good E2E test to write if we don't have HA API server tests?
what is needed for this issue to complete? Do you think you'll get that by Monday, which is Code Freeze?
We need some E2E testing or add an integration test as defined . It will not be done today.
so are you planning to move it to 1.11, then? It's about to be automatically kicked out of 1.10 by automation.
I do not have permission to change the milestone. If you could kindly change it, that would be awesome!
[MILESTONENOTIFIER] Milestone Removed From Issue Important: This issue was missing the label for more than 7 days. <details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Reopening as this was reverted in


Question: } volumeName := KubernetesVolumeID("aws://" + aws.StringValue(response.AvailabilityZone) + "/" + string(awsID)) <del> // AWS has a bad habbit of reporting success when creating a volume with // encryption keys that either don't exists or have wrong permissions. // Such volume lives for couple of seconds and then it's silently deleted // by AWS. There is no other check to ensure that given KMS key is correct, // because Kubernetes may have limited permissions to the key. if len(volumeOptions.KmsKeyID) > 0 { err := c.waitUntilVolumeAvailable(volumeName) if err != nil { if isAWSErrorVolumeNotFound(err) { err = fmt.Errorf("failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key") } return "", err </del> <ins> err = c.waitUntilVolumeAvailable(volumeName) if err != nil { // AWS has a bad habbit of reporting success when creating a volume with // encryption keys that either don't exists or have wrong permissions. // Such volume lives for couple of seconds and then it's silently deleted // by AWS. There is no other check to ensure that given KMS key is correct, // because Kubernetes may have limited permissions to the key. if isAWSErrorVolumeNotFound(err) { err = fmt.Errorf("failed to create encrypted volume: the volume disappeared after creation, most likely due to inaccessible KMS encryption key") </ins> } <ins> return "", err </ins> } return volumeName, nil

What happened: Kubernetes tries to attach EBS volumes while their status is still : I believe this started to happen after , when volumes and their tags started to be created in the same request. Before that, tags would be set through another call (i.e., ), which retries if the volume isn't . What you expected to happen: Kubernetes should try to attach the volume only when it's available. In order to do that, should make sure the volume is when it returns. How to reproduce it (as minimally and precisely as possible): Just create a dynamically provisioned EBS volume. /kind bug /sig storage /sig aws
CC
What is the downside of this? IIRC - even when we tagged separately, we used to have few cases where by the time attachment was attempted, volume was not available. May be tagging while creating the volume makes it worse but someone has to wait for volume to become available - either attach call or create volume call itself.
In worse case this causes first attach to fail - but controller will retry anyway.
I think it re-introduces
No - looking at events attached to the pod, it looks like volume eventually got attached. I think is unrelated.
It is related. If you create volume with invalid keys, AWS returns you a volume and in this case Kubernetes will create a PV for it. But then the volume silently disappears. The only way how to detect that is to wait for the volume to either get Available or disappear.
we are still waiting for volume to become available if volume has kms key -
+1 on this issue. I have seen this quite often on my v1.12 cluster too. So this is an issue even before
we are still waiting for volume to become available if volume has kms key - Oh, sorry for the noise.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
EBS CSI driver is already regardless of whether kms key is used or not. Any objections to do the same for in-tree?
I like it.


Question: pod.ObjectMeta.Annotations = map[string]string{ kubelettypes.ConfigSourceAnnotationKey: kubelettypes.FileSource, } <del> pod.Spec.PriorityClassName = systemCriticalPriorityName pod.Spec.Priority = &value </del> <ins> pod.Spec.PriorityClassName = scheduling.SystemNodeCritical </ins> framework.ExpectEqual(kubelettypes.IsCriticalPod(pod), true, "pod should be a critical pod") } else {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Setting PodPriority directly when adminssion Priority enable,the Pod is rejected. To fix this, we need to create new priority classes for these tests. Which jobs are failing: Which test(s) are failing: I0822 19:01:41.165] should be able to create and delete a critical pod [It] Since when has it been failing: Testgrid link: Reason for failure: because adminssion Priority enableIf the priority class is not found, the Pod is rejected. Anything else we need to know:
/area e2e-test-framework /sig testing /kind bug /priority important-soon
This has now changed to this error: /reopen
Reopened this issue. <detailsIn response to : This has now changed to this error: /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
For reference: testgrid link /milestone v1.17
is this still in progress?
pr has updated, PTAL.
looks like has to merge for this to close
/milestone v1.18


Question: type daemonSetsController struct { *DaemonSetsController <del> dsStore   cache.Store podStore  cache.Store nodeStore cache.Store </del> <ins> dsStore      cache.Store podStore     cache.Store nodeStore    cache.Store fakeRecorder *record.FakeRecorder </ins> } func newTestController(initialObjects ...runtime.Object) (*daemonSetsController, *fakePodControl, *fake.Clientset) {

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): DaemonSet FailedPlacement Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): OL7.3 Kernel (e.g. ): 4.1.12-94.2.1.el7uek.x86_64 Install tools: Others: What happened: I have a DaemonSet that is only deployed to a subset of nodes in the cluster based on a . I'm seeing events fired with reason for the daemonset pods on nodes that do not fit the selector. What you expected to happen: These events should never fire because they do not meet the node selectors in the pod spec. How to reproduce it (as minimally and precisely as possible): Create a cluster with different sized nodes. E.g. smaller masters than workers. Then deploy the daemonset only to the workers. You will see events triggered for the master nodes that don't meet the requirements. Anything else we need to know:
DaemonSet controller will run predicates to check if node should run daemon pod. In the progress of predicates, it will check if pod fits resources first before other predicates such as PodSelectorMatches. /cc
I think this bug appeared when critical pods were introduced and order of checking predicates changed. The event is useless for daemonsets in current situation :/ I'm not sure if there is a clean way to fix it.
I feel like a clean solution would be to look at the predicate reasons twice. The first time you look at the reasons and keep track if any are not intentional. If all of the reasons are intentional then skip checking the reasons.
actually just changing the order of checks in GeneralPredicates may just work. I'm testing the fix now.
Ok, GeneralPredicates returns all reasons. Error that pod doesn't match the node is also returned but because it's "intentional" there is no event for it.

The event "Node didn't have enough resource" should be sent only when there are no other "intentional" reasons.
Yes, I think maybe we need to check reason list twice: check is in tail of predicate list.
/cc
any PR for this? If not, I can help
I will have some time to work on it next week. If you have a spare time you can create a PR.
/assign


Question: R: &etcd.Response{ EtcdIndex: 10, Node: &etcd.Node{ <ins> Dir: true, </ins> Nodes: []*etcd.Node{ { <ins> Key:           "/foo", </ins> Value:         `{"id":"foo","kind":"Pod","apiVersion":"v1beta1"}`, ModifiedIndex: 1, }, { <ins> Key:           "/bar", </ins> Value:         `{"id":"bar","kind":"Pod","apiVersion":"v1beta1"}`, ModifiedIndex: 2, }, { <ins> Key:           "/baz", </ins> Value:         `{"id":"baz","kind":"Pod","apiVersion":"v1beta1"}`, ModifiedIndex: 3, }, { <ins> Key:   "/directory", </ins> Value: `{"name": "directory"}`, Dir:   true, },

kubectl returns resources in a random order, which makes it hard to look for things when users have a lot resources. Returning resources in deterministic order should be a helpful and trivial change (hopefully). We have an issue about pagination , but that's a larger and separate issue.
We should be cause to not do this on the * namespace - I think thats a special case. No disagreement for other list calls
What is the default etcd response order? Last created or last modified? Sent from my iPhone
I think the order is by key if sort param is given. If not given, the order is by last created (maybe, but we shouldn't trust it). It seems we are not trying to get sorted result from etcd.
Today, I think it's random order in the child key space in etcd (because the node structure uses a map), so you get golang map iteration order without sort. I don't know if sort on server is any cheaper than sort on client today, so we may want to do this in rest / registry rather than by passing etcd.
Apart from performance, sorting in clients means we can also take control of how responses are sorted, so for * namespace, we can sort by namespace, then by resource ID. , Clayton Coleman wrote: Master of Electrical and Computer Engineering Carnegie Mellon University
We can always hint sort to the registry storage layer in the future, so sorting on the client now won't kill us.
Just took a closer look at our rest/registry code to implement the sort (in client), but implementation-wise, it seems much easier to pass sort to etcd (one line change). Also, namespace is a prefix on resource, so sort across all namespace with etcd key will essentially sort resources by namespace then by Name. If we are to implement sort on client side, we can either: sort package: have each list object, ie.e PodList, ControllerList, implement sort interface. (We won't do this). have ListMeta implement sort interface, if there is a way to find enclosing structs (so we can use items field) have api.Object implement sort interface, verify object is list before sorting. sort algorithm ourself, and do it in rest. Is there other golang or kubernetes idom that I haven't been aware of that can be applied to this use case?

----- Original Message ----- I think sorting it on etcd is fine for now - that's the simplest possible thing that works.
Is there any material perf impact in etcd to enable it? Sent from my iPhone
Probably not material. Can always switch away if necessary
Famous last words :-) Sent from my iPhone
Got my hat ready


Question: } versionPath := pkgPath(group, version) <del> generator := pkg_runtime.NewDeepCopyGenerator(api.Scheme.Raw(), versionPath, sets.NewString("k8s.io/kubernetes")) </del> <ins> generator := kruntime.NewDeepCopyGenerator(api.Scheme.Raw(), versionPath, sets.NewString("k8s.io/kubernetes")) </ins> generator.AddImport(path.Join(pkgBase, "api")) if len(*overwrites) > 0 {

Forked from pointed out that all package names should be lower case, no under_scores or mixedCaps. The same rule applies to renaming imported packages. () generated a list of mixedCaps package names in our codebase: I will upload a PR later to rename kubeletTypes to kubetypes, and the rest is up for grabs.
sorry i have not saw this issue as i open a similar one(), do you mind i participate in this issue? i would happy to do some work for this. best regards.
of course, fixes are welcome. Thanks for picking up on this! We'll try to expedite the review and merge process of such PRs to avoid a lot of churns, but chances are that some rebases are needed :)
ok, i will send some fix sonner, and tag you for review. thanks again.
Not done:


Question: } func TestMakePorts(t *testing.T) { <del> var testCases = []struct { </del> <ins> var successTestCases = []struct { </ins> spec  string ports []api.Port }{

Kubernetes will create replication controller and pods in case of port spec error, e.g. will create 'myNginx'. A following correct spec will fail:
/cc


Question: iterations := 100 <ins> ginkgo.By("getting a starting resourceVersion") configmaps, err := c.CoreV1().ConfigMaps(ns).List(context.TODO(), metav1.ListOptions{}) framework.ExpectNoError(err, "Failed to list configmaps in the namespace %s", ns) resourceVersion := configmaps.ResourceVersion </ins> ginkgo.By("starting a background goroutine to produce watch events") donec := make(chan struct{}) stopc := make(chan struct{})

By inspection, the "should receive events on concurrent watches in same order" conformance test does two things that are not stable: Starts watches from resourceVersion=0 Relies on watch connections without handling timeouts/restarts This test can fail on clusters that have configmaps autocreated in namespaces already because ordering of watch events for resourceVersion=0 watches is undefined (see ) Watch connections used by conformance tests should use well-defined ordering (starting from resourceVersions other than 0) and handle interruptions properly. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } }) <del> ginkgo.It("should only allow access from service loadbalancer source ranges [Slow]", func() { // this feature currently supported only on GCE/GKE/AWS e2eskipper.SkipUnlessProviderIs("gce", "gke", "aws") loadBalancerCreateTimeout := e2eservice.GetServiceLoadBalancerCreationTimeout(cs) namespace := f.Namespace.Name serviceName := "lb-sourcerange" jig := e2eservice.NewTestJig(cs, namespace, serviceName) ginkgo.By("Prepare allow source ips") // prepare the exec pods // acceptPod are allowed to access the loadbalancer acceptPod := e2epod.CreateExecPodOrFail(cs, namespace, "execpod-accept", nil) dropPod := e2epod.CreateExecPodOrFail(cs, namespace, "execpod-drop", nil) ginkgo.By("creating a pod to be part of the service " + serviceName) // This container is an nginx container listening on port 80 // See kubernetes/contrib/ingress/echoheaders/nginx.conf for content of response _, err := jig.Run(nil) framework.ExpectNoError(err) // Make sure acceptPod is running. There are certain chances that pod might be teminated due to unexpected reasons. acceptPod, err = cs.CoreV1().Pods(namespace).Get(context.TODO(), acceptPod.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get pod %s", acceptPod.Name) framework.ExpectEqual(acceptPod.Status.Phase, v1.PodRunning) framework.ExpectNotEqual(acceptPod.Status.PodIP, "") // Create loadbalancer service with source range from node[0] and podAccept svc, err := jig.CreateTCPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.LoadBalancerSourceRanges = []string{acceptPod.Status.PodIP + "/32"} }) framework.ExpectNoError(err) defer func() { ginkgo.By("Clean up loadbalancer service") e2eservice.WaitForServiceDeletedWithFinalizer(cs, svc.Namespace, svc.Name) }() svc, err = jig.WaitForLoadBalancer(loadBalancerCreateTimeout) framework.ExpectNoError(err) ginkgo.By("check reachability from different sources") svcIP := e2eservice.GetIngressPoint(&svc.Status.LoadBalancer.Ingress[0]) // We should wait until service changes are actually propagated in the cloud-provider, // as this may take significant amount of time, especially in large clusters. // However, the information whether it was already programmed isn't achievable. // So we're resolving it by using loadBalancerCreateTimeout that takes cluster size into account. checkReachabilityFromPod(true, loadBalancerCreateTimeout, namespace, acceptPod.Name, svcIP) checkReachabilityFromPod(false, loadBalancerCreateTimeout, namespace, dropPod.Name, svcIP) // Make sure dropPod is running. There are certain chances that the pod might be teminated due to unexpected reasons. dropPod, err = cs.CoreV1().Pods(namespace).Get(context.TODO(), dropPod.Name, metav1.GetOptions{}) framework.ExpectNoError(err, "Unable to get pod %s", dropPod.Name) framework.ExpectEqual(acceptPod.Status.Phase, v1.PodRunning) framework.ExpectNotEqual(acceptPod.Status.PodIP, "") ginkgo.By("Update service LoadBalancerSourceRange and check reachability") _, err = jig.UpdateService(func(svc *v1.Service) { // only allow access from dropPod svc.Spec.LoadBalancerSourceRanges = []string{dropPod.Status.PodIP + "/32"} }) framework.ExpectNoError(err) // We should wait until service changes are actually propagates, as this may take // significant amount of time, especially in large clusters. // However, the information whether it was already programmed isn't achievable. // So we're resolving it by using loadBalancerCreateTimeout that takes cluster size into account. checkReachabilityFromPod(false, loadBalancerCreateTimeout, namespace, acceptPod.Name, svcIP) checkReachabilityFromPod(true, loadBalancerCreateTimeout, namespace, dropPod.Name, svcIP) ginkgo.By("Delete LoadBalancerSourceRange field and check reachability") _, err = jig.UpdateService(func(svc *v1.Service) { svc.Spec.LoadBalancerSourceRanges = nil }) framework.ExpectNoError(err) // We should wait until service changes are actually propagates, as this may take // significant amount of time, especially in large clusters. // However, the information whether it was already programmed isn't achievable. // So we're resolving it by using loadBalancerCreateTimeout that takes cluster size into account. checkReachabilityFromPod(true, loadBalancerCreateTimeout, namespace, acceptPod.Name, svcIP) checkReachabilityFromPod(true, loadBalancerCreateTimeout, namespace, dropPod.Name, svcIP) }) ginkgo.It("should be able to create an internal type load balancer [Slow]", func() { e2eskipper.SkipUnlessProviderIs("azure", "gke", "gce") createTimeout := e2eservice.GetServiceLoadBalancerCreationTimeout(cs) pollInterval := framework.Poll * 10 namespace := f.Namespace.Name serviceName := "lb-internal" jig := e2eservice.NewTestJig(cs, namespace, serviceName) ginkgo.By("creating pod to be part of service " + serviceName) _, err := jig.Run(nil) framework.ExpectNoError(err) enableILB, disableILB := enableAndDisableInternalLB() isInternalEndpoint := func(lbIngress *v1.LoadBalancerIngress) bool { ingressEndpoint := e2eservice.GetIngressPoint(lbIngress) // Needs update for providers using hostname as endpoint. return strings.HasPrefix(ingressEndpoint, "10.") } ginkgo.By("creating a service with type LoadBalancer and cloud specific Internal-LB annotation enabled") svc, err := jig.CreateTCPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeLoadBalancer enableILB(svc) }) framework.ExpectNoError(err) defer func() { ginkgo.By("Clean up loadbalancer service") e2eservice.WaitForServiceDeletedWithFinalizer(cs, svc.Namespace, svc.Name) }() svc, err = jig.WaitForLoadBalancer(createTimeout) framework.ExpectNoError(err) lbIngress := &svc.Status.LoadBalancer.Ingress[0] svcPort := int(svc.Spec.Ports[0].Port) // should have an internal IP. framework.ExpectEqual(isInternalEndpoint(lbIngress), true) // ILBs are not accessible from the test orchestrator, so it's necessary to use //  a pod to test the service. ginkgo.By("hitting the internal load balancer from pod") framework.Logf("creating pod with host network") hostExec := launchHostExecPod(f.ClientSet, f.Namespace.Name, "ilb-host-exec") framework.Logf("Waiting up to %v for service %q's internal LB to respond to requests", createTimeout, serviceName) tcpIngressIP := e2eservice.GetIngressPoint(lbIngress) if pollErr := wait.PollImmediate(pollInterval, createTimeout, func() (bool, error) { cmd := fmt.Sprintf(`curl -m 5 'http://%v:%v/echo?msg=hello'`, tcpIngressIP, svcPort) stdout, err := framework.RunHostCmd(hostExec.Namespace, hostExec.Name, cmd) if err != nil { framework.Logf("error curling; stdout: %v. err: %v", stdout, err) return false, nil } if !strings.Contains(stdout, "hello") { framework.Logf("Expected output to contain 'hello', got %q; retrying...", stdout) return false, nil } framework.Logf("Successful curl; stdout: %v", stdout) return true, nil }); pollErr != nil { framework.Failf("ginkgo.Failed to hit ILB IP, err: %v", pollErr) } ginkgo.By("switching to external type LoadBalancer") svc, err = jig.UpdateService(func(svc *v1.Service) { disableILB(svc) }) framework.ExpectNoError(err) framework.Logf("Waiting up to %v for service %q to have an external LoadBalancer", createTimeout, serviceName) if pollErr := wait.PollImmediate(pollInterval, createTimeout, func() (bool, error) { svc, err := cs.CoreV1().Services(namespace).Get(context.TODO(), serviceName, metav1.GetOptions{}) if err != nil { return false, err } lbIngress = &svc.Status.LoadBalancer.Ingress[0] return !isInternalEndpoint(lbIngress), nil }); pollErr != nil { framework.Failf("Loadbalancer IP not changed to external.") } // should have an external IP. gomega.Expect(isInternalEndpoint(lbIngress)).To(gomega.BeFalse()) ginkgo.By("hitting the external load balancer") framework.Logf("Waiting up to %v for service %q's external LB to respond to requests", createTimeout, serviceName) tcpIngressIP = e2eservice.GetIngressPoint(lbIngress) e2eservice.TestReachableHTTP(tcpIngressIP, svcPort, e2eservice.LoadBalancerLagTimeoutDefault) // GCE cannot test a specific IP because the test may not own it. This cloud specific condition // will be removed when GCP supports similar functionality. if framework.ProviderIs("azure") { ginkgo.By("switching back to interal type LoadBalancer, with static IP specified.") internalStaticIP := "10.240.11.11" svc, err = jig.UpdateService(func(svc *v1.Service) { svc.Spec.LoadBalancerIP = internalStaticIP enableILB(svc) }) framework.ExpectNoError(err) framework.Logf("Waiting up to %v for service %q to have an internal LoadBalancer", createTimeout, serviceName) if pollErr := wait.PollImmediate(pollInterval, createTimeout, func() (bool, error) { svc, err := cs.CoreV1().Services(namespace).Get(context.TODO(), serviceName, metav1.GetOptions{}) if err != nil { return false, err } lbIngress = &svc.Status.LoadBalancer.Ingress[0] return isInternalEndpoint(lbIngress), nil }); pollErr != nil { framework.Failf("Loadbalancer IP not changed to internal.") } // should have the given static internal IP. framework.ExpectEqual(e2eservice.GetIngressPoint(lbIngress), internalStaticIP) } }) // This test creates a load balancer, make sure its health check interval // equals to gceHcCheckIntervalSeconds. Then the interval is manipulated // to be something else, see if the interval will be reconciled. ginkgo.It("should reconcile LB health check interval [Slow][Serial]", func() { const gceHcCheckIntervalSeconds = int64(8) // This test is for clusters on GCE. // (It restarts kube-controller-manager, which we don't support on GKE) e2eskipper.SkipUnlessProviderIs("gce") e2eskipper.SkipUnlessSSHKeyPresent() clusterID, err := gce.GetClusterID(cs) if err != nil { framework.Failf("framework.GetClusterID(cs) = _, %v; want nil", err) } gceCloud, err := gce.GetGCECloud() if err != nil { framework.Failf("framework.GetGCECloud() = _, %v; want nil", err) } namespace := f.Namespace.Name serviceName := "lb-hc-int" jig := e2eservice.NewTestJig(cs, namespace, serviceName) ginkgo.By("create load balancer service") // Create loadbalancer service with source range from node[0] and podAccept svc, err := jig.CreateTCPService(func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeLoadBalancer }) framework.ExpectNoError(err) defer func() { ginkgo.By("Clean up loadbalancer service") e2eservice.WaitForServiceDeletedWithFinalizer(cs, svc.Namespace, svc.Name) }() svc, err = jig.WaitForLoadBalancer(e2eservice.GetServiceLoadBalancerCreationTimeout(cs)) framework.ExpectNoError(err) hcName := gcecloud.MakeNodesHealthCheckName(clusterID) hc, err := gceCloud.GetHTTPHealthCheck(hcName) if err != nil { framework.Failf("gceCloud.GetHttpHealthCheck(%q) = _, %v; want nil", hcName, err) } framework.ExpectEqual(hc.CheckIntervalSec, gceHcCheckIntervalSeconds) ginkgo.By("modify the health check interval") hc.CheckIntervalSec = gceHcCheckIntervalSeconds - 1 if err = gceCloud.UpdateHTTPHealthCheck(hc); err != nil { framework.Failf("gcecloud.UpdateHttpHealthCheck(%#v) = %v; want nil", hc, err) } ginkgo.By("restart kube-controller-manager") if err := e2ekubesystem.RestartControllerManager(); err != nil { framework.Failf("e2ekubesystem.RestartControllerManager() = %v; want nil", err) } if err := e2ekubesystem.WaitForControllerManagerUp(); err != nil { framework.Failf("e2ekubesystem.WaitForControllerManagerUp() = %v; want nil", err) } ginkgo.By("health check should be reconciled") pollInterval := framework.Poll * 10 loadBalancerPropagationTimeout := e2eservice.GetServiceLoadBalancerPropagationTimeout(cs) if pollErr := wait.PollImmediate(pollInterval, loadBalancerPropagationTimeout, func() (bool, error) { hc, err := gceCloud.GetHTTPHealthCheck(hcName) if err != nil { framework.Logf("ginkgo.Failed to get HttpHealthCheck(%q): %v", hcName, err) return false, err } framework.Logf("hc.CheckIntervalSec = %v", hc.CheckIntervalSec) return hc.CheckIntervalSec == gceHcCheckIntervalSeconds, nil }); pollErr != nil { framework.Failf("Health check %q does not reconcile its check interval to %d.", hcName, gceHcCheckIntervalSeconds) } }) </del> /* Release: v1.19 Testname: Service, ClusterIP type, session affinity to ClientIP

Which jobs are flaking: gce-master-scale-correctness Which test(s) are flaking: Services should be able to change the type and ports of a service [Slow] Testgrid link: https://k8s- Reason for failure: Test timeouts in ~half of the runs since it was enabled here: /sig scalability /sig network
/cc /cc
The context is - the problems are strictly related to scale - 5k-node jobs are timing out. I very briefly skimmed through the test and didn't see anything obviously suspicious (apart from the fact there are gazillions long timeouts there, which alltogether sum up long hours). That said - it seems to me that there are effectively two separate tests inside it: one for TCP one for UDP They don't seem to share anything with each other. - would it make sense to split this test two effectively two separate tests: one for TCP and one for UDP. (I guess it would also allow us to share some code there).
[And potentially it could help with debugging the problem if it is strictly related to only one of those cases...]
I wanted to focus on this past week but I didn´t have time, this is my list of more expensive times and approx average duration, all of the sig-network ones use LoadBalancers. I wanted to time the test to understand if there are places where can save time ... I don´t have much context on those tests, maybe Tim can help us more, but this comment makes me think that the use of both protocols is intended
/triage accepted
/triage accepted It's been a very long time since I touched this, and I recall holding my nose on it in the first place. Splitting TCP from UDP seems reasonable if we can run them in parallel. we'll want to share the flow, and parameterize some shared funcs. I don't know that this will prevent test timeouts, though.
We could use a volunteer to go poke at this, and see how hard it will really be.
You mean "forcing them to run in parallel"? Or "allowing them to run in parallel"? The later doesn't require any additiona work. The former I think mean that it will still have to be a single test (just the code will be organized better) and that won't really help with debugging (nor preventing timeouts).

this was discussed yesterday in sig-net meeting and my understanding was to split current test in 2 different so the test can run in parallel, if you want to do so and one protocol doesn´t have to wait for the other. I think that is what you meant here
As long as we CAN parallelize. Creating LBs can be sloooooow. :P Antonio Ojea <:
Reopening - I don't think just splitting will address the problem. It may help us with debugging though...
With the split we moved from 60 mins to run the test, to run 2 tests in [30-40] mins each https://k8s- Seems a win :smile:
Awesome - that's something I was silently counting for TBH :) [I doubt it will fix flakiness, but maybe it will make it easier to debug...]
Actually - I think we're getting close thanks to the two changes ( + this one). Here is the run: This particular one filed on UDP (not sure if it's just UDP-related, or just manifested with UDP now). And we have a clear failure now: In my PR I decreased the timeout from 2h to 45m, as it seemed to me that in the past it should be enough. But it failed on 45m timeout here. So I think the next question now is: is it that slow - that we need more than 45m or something isn't working as expected (either in the k8s itself or in the test) But I think we have a clear single place to look at now.
Actually - the full stacktrace: So it's this line that failed: What is interesting is that something was changing during those 45 minutes:
Note that it's 4th time we actually wait for LB in this test. The failure happens after the pod were scaled down to 0 and then to 1. So that makes me think, that's it's actually not LB-related, but rather something internally in k8s, becuase L4 LB doesn't depend on pods - it targets nodes...
If it is always UDP, I start to see ghosts between GCE and UDP ... we only have KIND to compare against, or do we have some other non gce enviroment?

I mean, there are other tests failing in UDP , and show much more failures in GCE than in kind
- this does not look "normal". Can we give some prio to helping track this flake?
hmm interesting. we will investigate.
/assign
- any progress on this?
I haven't had a chance to look into this yet. I will prioritize it this week.
Digging through the logs I believe this was a bug in the test code that was fixed by . In the test, the service port changes around 12:33. However due to the bug, the test continues to use the old service port as seen in the test output: The port should be 81. I believe the following sequence occurs from this point: 1) The reachability test fails in the beginning because the service has changed the port and the load balancer has not been updated yet resulting in the error. 1) Around 12:38, the service controller is syncing the service and notices that the service port has changed and starts to delete the forwarding rule causing the error to change to . 1) Once the forwarding rule is recreated, the error goes back to being because the forwarding rule exists but the wrong port is used. Looking at , the test runs after the one linked have all passed. Is there anything else that needs to be investigated as part of this issue?
OK - the test was quite flaky in the past but indeed it didn't flake for the last 3 weeks (neither this nor TCP one). I'm fine with closing this one - we can open a new one if this happens again. - thanks for debugging. - it seems that the splitting of the test actually made things waay better. Which is great.


Question: } total := dirInfo1.Size() + dirInfo2.Size() + fileInfo1.Size() + fileInfo2.Size() <del> size, err := DiskUsage(dir1) </del> <ins> usage, err := DiskUsage(dir1) </ins> if err != nil { t.Fatalf("TestDiskUsage failed: %s", err.Error()) } <ins> size, err := resource.ParseQuantity(fmt.Sprintf("%d", usage.Bytes)) if err != nil { t.Fatalf("TestDiskUsage failed: %s", err.Error()) } </ins> used, err := resource.ParseQuantity(fmt.Sprintf("%d", total)) if err != nil { t.Fatalf("TestDiskUsage failed: %s", err.Error())

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: in calculates number of inodes wrong when there are hard links. What you expected to happen: inodes with hardlinks should only be counted once. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Issue is still not solved. There is an open PR waiting for review. /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
still waiting for to be merged

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to run pods on Alpine Linux with busybox find. What you expected to happen: I expected the call to exit with success, not failure. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing GNU findutils, so busybox implementation of is used. Anything else we need to know?: failed because the argument is not implemented in busybox . The option is not specified in so it can not be expected to be supported in all implementations. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related bug Others: I think it would be better to use instead of Related to
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This is solved by which is waiting for a review. /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Still waiting for the PR to be merged. /remove-lifecycle rotten
/remove-lifecycle rotten

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running test suite on a cluster with workers on Alpine Linux, which uses busybox instead of GNU coreutils fails. The log shows that it fails to execute the command. What you expected to happen: I expected the call to exit with success. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing . Anything else we need to know?: It failed due to the option which is not implemented in busybox. The option is not specified in so I don't think it is reasonable to expect it to implemented in busybox. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related Others:
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale fixes this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten
Still waiting for PR to be merged


Question: // GetSELinuxSupport returns true if given path is on a mount that supports // SELinux. func (hu *HostUtil) GetSELinuxSupport(pathname string) (bool, error) { <del> return GetSELinux(pathname, procMountInfoPath) </del> <ins> return GetSELinux(pathname, procMountInfoPath, selinux.SELinuxEnabled) </ins> } // GetOwner returns the integer ID for the user and group of the given path

On a busy system, kubelet failed to tell the container runtime that a pod's CSI volume supports SELinux and should be relabeled with pod's SELinux label. The pod then can't access the volume. Kubelet correctly detects that the CSI volume supports SELinux and tells the container runtime to relabel it. Unknown. Get a busy system and try hard? CSI volume plugin tried to detect that the volume supports SELinux during SetupAt. This check failed: Current CSI volume plugin code silently consumes the error and assumes that the volume does not support SELinux at all: Therefore, the volume is marked as mounted and given to the container runtime without the right SELinux options and the runtime does not relabel volume. <details22.0 </details<detailsNone </details<details</details<details</details<detailsCRI-O, but I think it's not really relevant. </details<details</details
/sig storage /triage accepted
We've updated kubelet in our clusters to v1.20.13 (which contains the cherry picked fix), but still observe the issue


Question: } cfg.NodeCIDRMaskSize = o.NodeCIDRMaskSize <ins> cfg.NodeCIDRMaskSizeIPv4 = o.NodeCIDRMaskSizeIPv4 cfg.NodeCIDRMaskSizeIPv6 = o.NodeCIDRMaskSizeIPv6 </ins> return nil }

What happened: I used dual-stack and set the parameter of kube-controller to --node-cidr-mask-size-ipv4=24 --node-cidr-mask-size-ipv6=96, but the podCidr my node got was still a /64 mask What you expected to happen: The mask the node gets should be the /96 I set How to reproduce it (as minimally and precisely as possible): Deploy the cluster using kubeadm, which is configured as followsAnd modify the network-plugin of kubelet to be kubenet Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: KVM virtual machine OS (e.g: ): VERSION="18.04.2 LTS (Bionic Beaver)" Kernel (e.g. ): Linux k8s02-01 4.15.0-70-generic -Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig network These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
is the configuration above correct? should be using integers instead of strings to configure the node-cidr-masks? Ref.
I initially tried using integers, but kubeadm init reported an error.That's why I replaced it with a string, and then I didn't get an error
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
thank you for reporting this. I'll take a look.
/assign


Question: // Returns a cache.ListWatch that finds all pods that need to be // scheduled. func (factory *ConfigFactory) createUnassignedPodLW() *cache.ListWatch { <del> return cache.NewListWatchFromClient(factory.Client, "pods", api.NamespaceAll, fields.Set{getHostFieldLabel(factory.Client.APIVersion()): ""}.AsSelector()) </del> <ins> return cache.NewListWatchFromClient(factory.Client, "pods", api.NamespaceAll, fields.Set{client.PodHost: ""}.AsSelector()) </ins> } func parseSelectorOrDie(s string) fields.Selector {

My broke it for v1beta3, then fixed it for v1beta3 and broke it for v1beta1. I'm working on a fix.
Honestly I'm very surprised that more people aren't complaining-- integration test logs are full of noise because of this.
Exact text of error log is "Failed to watch *api.Node: field label not supported: "
cc


Question: visibility = ["//visibility:public"], deps = [ "//pkg/api/legacyscheme:go_default_library", <ins> "//pkg/apis/admission/install:go_default_library", "//pkg/apis/admissionregistration/install:go_default_library", "//pkg/apis/apiserverinternal/install:go_default_library", </ins> "//pkg/apis/apps/install:go_default_library", "//pkg/apis/authentication/install:go_default_library", "//pkg/apis/authorization/install:go_default_library",

What happened: Tried to convert a v1beta1 Ingress object to v1, and received the following error: What you expected to happen: The conversion to work. How to reproduce it (as minimally and precisely as possible): Then run: /sig cli /assign
Can i work on this issue?
/assign
/triage accepted


Question: REGISTRY?=gcr.io/google-containers IMAGE?=debian-hyperkube-base <del> TAG=0.5 </del> <ins> TAG=0.6 </ins> ARCH?=amd64 CACHEBUST?=1

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Pods with volumeMount gitRepo volumes in 1.8.0 see the following error in kubelet logs (also in ) It is important to not that the git repo clone is done via ssh. I think what is happening is that hyperkube image now used to have command in it and now it does not have it. So the git clone command that kubelet runs fails with the above stacktrace since it cannot find . What you expected to happen: gitRepo is volume mounted. How to reproduce it (as minimally and precisely as possible): Reproducible in kubernetes 1.8.0+ hyperkube images. Anything else we need to know?: This probably is due to missing ssh binary in the new 1.8.0 hyperkube images. The commit might be an issue. Will comment further if I find otherwise. Environment: Kubernetes version (use ): =1.8.0 Cloud provider or hardware configuration**: Not Application. OS (e.g. from /etc/os-release): hyperkube base image Kernel (e.g. ): Install tools: Others:
How did you installed Kubernetes may I ask?
how does this help?
Normally, it is suggest to install kubelet as a daemon on host instead of using hyperkube container. cc you may be interested in this issue.
Need your help in identifying the fix here. I can send something for review. Suggestions are welcome.
The problem is that ssh isn't installed inside of the container. I don't know if it should be either, as the hyperkube image grows so large. If you install the openssh-client, it should work
Yes I did identify that as the problem if you see the description I mentioned. I pointed to a git commit that might have caused this. I am not sure what package installation has changed in the hyperkube container. "ssh" used to be available and now it is not. For the gitrepo feature to work it should be available.
Ah okay. If it was there before, we might want to add it back. SGTM.
Yeah, I switched the hyperkube image base from to our with some additional dependencies back. Apparently due to lack of volume + hyperkube test coverage, I've missed a number of dependencies needed.

cc
we could use dropbear as a replacement for just the ssh client?
I would prefer to use the openssh client over the dropbear client. It has wider usage and a stronger security track record IMO. The dropbear client also has somewhat different usage and configuration, so arguably using it is still a regression over the previous state. Saving a couple MB isn't worth using worse utilities, however much fun "dockerfile-golfing" is.
to fix
cherrypick to release-1.8:
This should be fixed in v1.8.3.


Question: "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//vendor/k8s.io/apimachinery/pkg/apis/meta/v1/unstructured:go_default_library", "//vendor/k8s.io/apimachinery/pkg/runtime:go_default_library", <ins> "//vendor/k8s.io/apimachinery/pkg/types:go_default_library", </ins> "//vendor/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//vendor/k8s.io/apimachinery/pkg/watch:go_default_library", "//vendor/k8s.io/apiserver/pkg/util/feature:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CRD: Existing object: New object: says: If this is a problem with the new object definition rather than a bug with , I'd expect a better error message. (The new object looks fine to me, though; it should be identical to the old object.) What you expected to happen: I expected the object to be updated without an error. How to reproduce it (as minimally and precisely as possible): See description. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: GCE OS (e.g. from /etc/os-release): CentOS Linux 7.4 (1708) Kernel (e.g. ): Linux master-head-1 4.13.4- SMP Wed Sep 27 13:32:23 EDT 2017 x8664 x8664 x8664 GNU/Linux Install tools: Custom scripts Others:
/sig api-machinery
Are you able to reproduce this everytime? I am able to use apply normally with your example.
I can reliably reproduce it if the annotation exists: If I use to remove the annotation, then it works the first time but fails thereafter:
This is happening for other CRDs as well (Prometheus Operator ServiceMonitors). If I apply a ServiceMonitor CRD 3 times: If I remove the annotation or replace the object () then I can work around the issue.
Seems any CRD is affected; the CRD used by our are also not working: This has completely broken our CI/CD workflow... unfortunately we'd already completed the TPR-CRD migration for the 1.8 upgrade before noticing this problem, so we can't really downgrade. How safe is as an alternative to ? It still generates an update event for watchers, not a delete/recreate, right?
It looks like something going wrong during the patching. After using changes from , I get this: /cc /area third-party-resource
But uses json merge patch and not strategic merge patch for custom resources:
I was able to reproduce this in v1.8.0 with the original manifests at the top. As found, it only happens on the 3rd execution: It also seems to only happen when the ought to be a no-op.

If I change something in the IPPool spec, works once more, then starts failing again. When the is a no-op, the PATCH request body generated by looks like this: When the is NOT a no-op, the PATCH request body looks like this:
There's a typo in the code generating the error messages for both and : They should be calling instead of . When I fix that, the error message becomes:
I can reproduce this without , just by doing a no-op twice:
The same issue affects v1.7.8 (tested on GKE)
Same issue with v1.8.1
This just fix the error message: But it doesn't fix the issue
Still investigating the issue but I noticed that when we do a no-op patch for native resources, we get a "not patched" response. However, for non-native resources, it still tries to patch it (and bumps the resourceVersion too).
I don't konwn whether this is a issue, I can't do strategic merge patch to CRD. When I run , --type=merge must be passed.
We should be able to apply strategic merge patch to CRDs. However, we can't apply strategic merge to CRs because we're missing the metadata about how to handle each field for the strategic merge patch. Unfortunately, this error is occuring even with . :(
/assign
do you know if the fix fir this will make it to 1.8.3
hopefully yes, this should be merged soon.
thanks! this has been a blocker for us to upgrade to 1.8
can you confirm this is completely fixed? The PR writes "partial fix".
Now we can't reproduce this issue with the method descripted above. I still need some I extra time to research it. Maybe we can close this one. If I can reproduce the issue in another way, I can open another issue with new situation.
If it's not fixed, let's keep it open.
Fun, a fork closing issues.
From Then I think we can officially close this.
(the fix is merged into release-1.8, but after v1.8.4-beta.0)
what stable release of k8s will this be in? 1.8.4 or 1.8.5?
Is there an update for this?

looks like it was merged here: but did not make it into 1.8.4, might make it into 1.8.5


Question: for _, initDriver := range csiTestDrivers { curDriver := initDriver(config) <ins> curConfig := curDriver.GetDriverInfo().Config </ins> Context(testsuites.GetDriverNameWithFeatureTags(curDriver), func() { <del> driver := curDriver </del> BeforeEach(func() { <ins> // Reset config. The driver might have modified its copy // in a previous test. curDriver.GetDriverInfo().Config = curConfig </ins> // setupDriver <del> driver.CreateDriver() </del> <ins> curDriver.CreateDriver() </ins> }) AfterEach(func() { // Cleanup driver <del> driver.CleanupDriver() </del> <ins> curDriver.CleanupDriver() </ins> }) <del> testsuites.RunTestSuite(f, driver, csiTestSuites, csiTunePattern) </del> <ins> testsuites.RunTestSuite(f, curDriver, csiTestSuites, csiTunePattern) </ins> }) }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: gce/gke multizone/regional jobs Which test(s) are failing: Only PD tests Since when has it been failing: Around 12/21 with the 16:49 run Testgrid link: https://k8s- Reason for failure: TBD Anything else we need to know: <!-- DO NOT EDIT BELOW THIS LINE --/kind failing-test
The runs started flaking after merged. cc Here's one failing run: Pod pod-subpath-test-gcepd-dynamicpv-pfsm failed to schedule because of conflicting node selectors: We shouldn't need to set the node selector for PD tests using PVs.
The PR broke the subpath test because the modification to the config struct embedded in the driver info now persists across tests: A quick fix would be to reset that field in CreateDriver. The long-term fix is the change discussed in My preference is to do the quick fix now, then either merge or close PRs in this order: (needs a decision about logging of tests that aren't getting defined, see ) a future PR for What I'd like to avoid is having to rebase PR on top of the long-term solution for issue - that'll be lots of code conflicts.
/assign PR is here: PR


Question: return &REST{store} } <ins> // Implement CategoriesProvider var _ rest.CategoriesProvider = &REST{} // Categories implements the CategoriesProvider interface. Returns a list of categories a resource is part of. func (c *REST) Categories() []string { return []string{"api-extensions"} } </ins> var swaggerMetadataDescriptions = metav1.ObjectMeta{}.SwaggerDoc() // ConvertToTable implements the TableConvertor interface for REST.

What would you like to be : Adding a "webhooks" group to kubectl to show mutating/validating webhooks would be super useful Why is this needed: There are a few reasons this is useful (to me): often cannot remember the right names for these resources, are very long and I often have a typo often want to enumerate all of them on the system. cc
/sig cli
I think this is a great idea, definitely something I could've used in the past. I'd be interested in taking a pass at implementing this if the proposal gets accepted.
Instead of building a category based on the types we have, I would build the category based on why people look at them. I suspect a category like which returned admission config admission config for various CSI types to list in CRD would be very helpful.
Webhook is far too generic. By putting admission webhooks in there we basically steal that word from the ecosystem to be used for other purposes. "admissionwebhooks" I could agree with.
Yeah, there are other webhooks, e.g. CRD conversion webhooks or authz webhooks...
I'd expect the short names "mwc" and "vwc" to be set up, are they not?
Unfortunately, they are not.
/triage accepted /assign
does category tolerate namespacing? i.e. would be tolerated? Still a bit long, but shorter than
It's a string, so you could theoretically namespace that.


Question: ns := f.Namespace.Name By("Initializing PV source") <del> t.pvSource = t.createGCEVolume() </del> <ins> t.pvSource, _ = framework.CreateGCEVolume() </ins> pvConfig := framework.PersistentVolumeConfig{ NamePrefix: "pv-upgrade", PVSource:   *t.pvSource,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --We should have an e2e to test issue . The test needs to delete a PVC while a pod is still using it, and then restart the controller manager. /kind bug /sig storage
CC
/assign unofficially so I can find this later.


Question: active := int32(len(jobCtx.activePods)) newSucceededPods, newFailedPods := getNewFinishedPods(jobCtx) jobCtx.succeeded = job.Status.Succeeded + int32(len(newSucceededPods)) + int32(len(jobCtx.uncounted.succeeded)) <del> failed := job.Status.Failed + int32(len(newFailedPods)) + int32(len(jobCtx.uncounted.failed)) </del> <ins> failed := job.Status.Failed + int32(nonIgnoredFailedPodsCount(jobCtx, newFailedPods)) + int32(len(jobCtx.uncounted.failed)) </ins> var ready *int32 if feature.DefaultFeatureGate.Enabled(features.JobReadyPods) { ready = pointer.Int32(countReadyPods(jobCtx.activePods))

Pod backoff calculation excludes pods that match an rule in the failure policy: This could lead to fast retries, overwhelming the control plane, if most failures match an rule. The semantics of is not to count towards the limit, but the backoff should still be respected. /sig apps Recreations to be throttled. a Job with failure policy that matches every kind of failure. a container that always fails You would observe immediate recreation. No response_ 27+ <detailsN/A </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package filters import ( "net/http" "net/http/httptest" "testing" ) func TestCacheControl(t *testing.T) { tests := []struct { name string path string startingHeader string expectedHeader string }{ { name:           "simple", path:           "/api/v1/namespaces", expectedHeader: "no-cache, private", }, { name:           "openapi", path:           "/openapi/v2", expectedHeader: "no-cache, private", }, { name:           "already-set", path:           "/api/v1/namespaces", startingHeader: "nonsense", expectedHeader: "nonsense", }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { handler := http.HandlerFunc(func(http.ResponseWriter, *http.Request) { //do nothing }) wrapped := WithCacheControl(handler) testRequest, err := http.NewRequest(http.MethodGet, test.path, nil) if err != nil { t.Fatal(err) } w := httptest.NewRecorder() if len(test.startingHeader) > 0 { w.Header().Set("Cache-Control", test.startingHeader) } wrapped.ServeHTTP(w, testRequest) actual := w.Header().Get("Cache-Control") if actual != test.expectedHeader { t.Fatal(actual) } }) } } </ins>

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: // WriteHeader implements http.ResponseWriter. func (rl *respLogger) WriteHeader(status int) { <ins> rl.recordStatus(status) rl.w.WriteHeader(status) } // Hijack implements http.Hijacker. func (rl *respLogger) Hijack() (net.Conn, *bufio.ReadWriter, error) { rl.hijacked = true return rl.w.(http.Hijacker).Hijack() } func (rl *respLogger) recordStatus(status int) { </ins> rl.status = status <ins> rl.statusRecorded = true </ins> if rl.logStacktracePred(status) { // Only log stacks for errors stack := make([]byte, 2048)

From kube-, Kube-apiserver returns 200 as Success code upon some requests, otherwise return 0 as Success. I1102 20:11:33. 5 ] GET /api/v1/nodes/kubernetes-minion-3io1: (2.683448ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.5:] I1102 20:11:33. 5 ] PUT /api/v1/nodes/kubernetes-minion-3io1/status: (5.474235ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.5:] I1102 20:11:34. 5 ] GET /api/v1/nodes: (2.139174ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:37. 5 ] GET /api/v1/nodes/kubernetes-minion-jmxr: (1.452673ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.3:] I1102 20:11:37. 5 ] PUT /api/v1/nodes/kubernetes-minion-jmxr/status: (4.833358ms) 200 [[kubelet/v1.1.1 (linux/amd64) kubernetes/] 10.240.0.3:] I1102 20:11:38. 5 ] GET /healthz: (104.272µs) 0 [[Go 1.1 package http] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumes?resourceVersion=1687: (9.998412938s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39.

5 ] GET /api/v1/watch/persistentvolumeclaims?resourceVersion=1687: (10.000876825s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/persistentvolumes?resourceVersion=1687: (9.998155747s) 0 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:39. 5 ] GET /api/v1/watch/namespaces?resourceVersion=1667: (59.977954152s) 0 [[heapster/v1.0.3 (linux/amd64) kubernetes/$Format] 10.240.0.5:] I1102 20:11:39. 5 ] GET /api/v1/nodes: (1.471292ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumes: (884.225µs) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumeclaims: (942.355µs) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/persistentvolumes: (1.065416ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:] I1102 20:11:40. 5 ] GET /api/v1/namespaces: (1.441822ms) 200 [[heapster/v1.0.3 (linux/amd64) kubernetes/$Format] 10.240.0.5:] I1102 20:11:40.

5 ] GET /api/v1/nodes: (1.958739ms) 200 [[kube-controller-manager/v1.1.1 (linux/amd64) kubernetes/] 127.0.0.1:]
If it were only the watches I would understand, but it's pretty weird for /healthz.
Looking into this, this is likely a false alarm due to an error in pkg/httplog. It assumes that everything calls does. In this case, the status is defaulted to 200. From the So when you just write, the httplog package doesn't see the status and assumes it's 0. I'm going to work on reproducing, confirming, then fixing this.
I confirmed with curl that this is just a logging error at least for healthz. Status code actually returns was 200. This is also pretty obvious since someone writing a 0 status code would result in a stacktrace in the log.
Watches are a different issue than healthz. I think watches are hijacked, and that kinda disables all status recording ability by this package. Do we care to fix that?
websockets hijack-- I thought watches did ordinary HTTP chunked encoding? , krousey wrote:
Probably, but the way it's currently implemented, hijacking completely bypasses this middle-ware logging layer we have.
Maybe we detect hijacking and log that instead of false data? e.g. would become
works for me! , krousey wrote:
Ok out for review in


Question: // Represents a projected volume source type ProjectedVolumeSource struct { // list of volume projections <ins> // +optional </ins> Sources []VolumeProjection `json:"sources" protobuf:"bytes,1,rep,name=sources"` // Mode bits used to set permissions on created files by default. // Must be an octal value between 0000 and 0777 or a decimal value between 0 and 511.

What happened: I created a Pod with a ProjectedVolume with an empty array of , but Kubernetes transformed that value to . Some clients are happy with either value (e.g. Go) while others are not (e.g. Python). For example, the following is accepted and stored by Kubernetes, but cannot be read by the Python client: The field has been transformed to : Even a JSON patch has no effect: What you expected to happen: I expected the required (OpenAPI) field of core/v1.ProjectedVolumeSource to be returned as the empty array I submitted. How to reproduce it (as minimally and precisely as possible): Create the above Pod: See that has been transformed to : See that the officially-supported Python client is unable to deserialize the Pod: Anything else we need to know?: I recognize there's a lot of layers involved in the steps above. The ineffectiveness of the JSON patch is what makes me think the culprit is somewhere in the field spec or API somehow. /sig api-machinery Environment: Kubernetes version (use ): [official-clients]:
/remove-sig api-machinery /sig storage the issue is in the API definition, which does not indicate the sources field is optional (even though it is)
/cc