Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package options import ( "fmt" "io/ioutil" "net/http" "net/http/httptest" "os" "path/filepath" "strings" "testing" "k8s.io/kubernetes/pkg/apis/componentconfig" ) func TestSchedulerOptions(t *testing.T) { // temp dir tmpDir, err := ioutil.TempDir("", "scheduler-options") if err != nil { t.Fatal(err) } defer os.RemoveAll(tmpDir) // record the username requests were made with username := "" // https server server := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { username, _, _ = req.BasicAuth() if username == "" { username = "none, tls" } w.WriteHeader(200) w.Write([]byte(`ok`)) })) defer server.Close() // http server insecureserver := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) { username, _, _ = req.BasicAuth() if username == "" { username = "none, http" } w.WriteHeader(200) w.Write([]byte(`ok`)) })) defer insecureserver.Close() // config file and kubeconfig configFile := filepath.Join(tmpDir, "scheduler.yaml") configKubeconfig := filepath.Join(tmpDir, "config.kubeconfig") if err := ioutil.WriteFile(configFile, []byte(fmt.Sprintf(` apiVersion: componentconfig/v1alpha1 kind: KubeSchedulerConfiguration clientConnection: kubeconfig: "%s" leaderElection: leaderElect: true`, configKubeconfig)), os.FileMode(0600)); err != nil { t.Fatal(err) } if err := ioutil.WriteFile(configKubeconfig, []byte(fmt.Sprintf(` apiVersion: v1 kind: Config clusters: - cluster: insecure-skip-tls-verify: true server: %s name: default contexts: - context: cluster: default user: default name: default current-context: default users: - name: default user: username: config `, server.URL)), os.FileMode(0600)); err != nil { t.Fatal(err) } // flag-specified kubeconfig flagKubeconfig := filepath.Join(tmpDir, "flag.kubeconfig") if err := ioutil.WriteFile(flagKubeconfig, []byte(fmt.Sprintf(` apiVersion: v1 kind: Config clusters: - cluster: insecure-skip-tls-verify: true server: %s name: default contexts: - context: cluster: default user: default name: default current-context: default users: - name: default user: username: flag `, server.URL)), os.FileMode(0600)); err != nil { t.Fatal(err) } // Insulate this test from picking up in-cluster config when run inside a pod // We can't assume we have permissions to write to /var/run/secrets/... from a unit test to mock in-cluster config for testing originalHost := os.Getenv("KUBERNETES_SERVICE_HOST") if len(originalHost) > 0 { os.Setenv("KUBERNETES_SERVICE_HOST", "") defer os.Setenv("KUBERNETES_SERVICE_HOST", originalHost) } testcases := []struct { name             string options          *Options expectedUsername string expectedError    string }{ { name:             "config file", options:          &Options{ConfigFile: configFile}, expectedUsername: "config", }, { name: "kubeconfig flag", options: &Options{ ComponentConfig: componentconfig.KubeSchedulerConfiguration{ ClientConnection: componentconfig.ClientConnectionConfiguration{ KubeConfigFile: flagKubeconfig}}}, expectedUsername: "flag", }, { name:             "overridden master", options:          &Options{Master: insecureserver.URL}, expectedUsername: "none, http", }, { name:          "no config", options:       &Options{}, expectedError: "no configuration has been provided", }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { // create the config config, err := tc.options.Config() // handle errors if err != nil { if tc.expectedError == "" { t.Error(err) } else if !strings.Contains(err.Error(), tc.expectedError) { t.Errorf("expected %q, got %q", tc.expectedError, err.Error()) } return } // ensure we have a client if config.Client == nil { t.Error("unexpected nil client") return } // test the client talks to the endpoint we expect with the credentials we expect username = "" _, err = config.Client.Discovery().RESTClient().Get().AbsPath("/").DoRaw() if err != nil { t.Error(err) return } if username != tc.expectedUsername { t.Errorf("expected server call with user %s, got %s", tc.expectedUsername, username) } }) } } </ins>

/kind bug /sig scheduling What happened: kube-scheduler that is pointed to a componentconfig file having ignores that it should use this kubeconfig file. Instead, it uses the in-cluster config (as is neither provided). What you expected to happen: The kube-scheduler should respect the kubeconfig provided in the . How to reproduce it (as minimally and precisely as possible): Starting image with whereby the content of looks like this: makes the scheduler logging The exact same configuration as above works with branch. Seems like has introduced the behaviour. The kube clients are created but the componentconfig file is loaded . /cc Environment: Kubernetes version (use ): 1.11.0-rc.2
/milestone v1.11 for triage
Could you please take a look at this? It seems related to the changes you made recently.
/assign
/assign /priority critical-urgent bumping priority since this would be a 1.11 regression
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: package network import ( <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> networkingv1 "k8s.io/api/networking/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr"

What happened: I wanted to run the e2e network policy tests with: and I noticed that all tests are failing (I actually don't know why nobody has noticed this before). Looking into the logs I saw the following errors: which are correct because the tests using as "client" which doesn't contain . What you expected to happen: The e2e tests for network policies should actually pass. How to reproduce it (as minimally and precisely as possible): Just run: Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: kind OS (e.g: ): Kernel (e.g. ): Install tools: Others:
/sig testing
With the current fix two tests are still failing: I will take a look and fix these failing tests too.
Seems to be an implementation error on the network plugin side (tested with weave latest and calico 3.6). I will investigate some time in the implementation details on the calico side. But this error/failures are no blocker for the PR from my perspective.

As far as I know the busybox image doesn't contain , only so I don't understand how the PR that changed from to has passed the CI What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug cc
/sig network
NetworkPolicy tests are a and so don't get run by default. But I imagine at least some of the CI test environments implement NetworkPolicy (e2e-gce?) so probably we should fix that. So 2 fixes: Make the tests pass again; probably requires changing the test image from busybox to something else. I bet there are other tests in that they switched from wget to curl where they did remember to change the image (because the tests would have failed otherwise) so this should just be a matter of seeing what image they used there, and use it here too. Make at least one of the CI tests test NetworkPolicy /good-first-issue (at least the first part anyway)
/remove-good-first-issue /remove-help (after someone asked me about this, I realized that trying to run tests that don't get run by CI is kind of complicated...)


Question: c.queue.Add(key) } <del> // ensureSetupManagedByAnnotation selects all EndpointSlices for a Service and // ensures they have LabelManagedBy set appropriately. This ensures that only // one controller or entity is trying to manage a given EndpointSlice. This // function provides backwards compatibility with the initial alpha release of // EndpointSlices that did not include these labels. // TODO(robscott): Remove this in time for v1.18. func (c *Controller) ensureSetupManagedByAnnotation(service *v1.Service) error { if managedBySetup, ok := service.Annotations[managedBySetupAnnotation]; ok && managedBySetup == managedBySetupCompleteValue { return nil } esLabelSelector := labels.Set(map[string]string{discovery.LabelServiceName: service.Name}).AsSelectorPreValidated() endpointSlices, err := c.endpointSliceLister.EndpointSlices(service.Namespace).List(esLabelSelector) if err != nil { c.eventRecorder.Eventf(service, v1.EventTypeWarning, "FailedToListEndpointSlices", "Error listing Endpoint Slices for Service %s/%s: %v", service.Namespace, service.Name, err) return err } for _, endpointSlice := range endpointSlices { if _, ok := endpointSlice.Labels[discovery.LabelManagedBy]; !ok { if endpointSlice.Labels == nil { endpointSlice.Labels = make(map[string]string) } endpointSlice.Labels[discovery.LabelManagedBy] = controllerName _, err = c.client.DiscoveryV1beta1().EndpointSlices(endpointSlice.Namespace).Update(endpointSlice) if err != nil { return err } } } if service.Annotations == nil { service.Annotations = make(map[string]string) } service.Annotations[managedBySetupAnnotation] = managedBySetupCompleteValue _, err = c.client.CoreV1().Services(service.Namespace).Update(service) return err } </del> func (c *Controller) addPod(obj interface{}) { pod := obj.(*v1.Pod) services, err := c.serviceSelectorCache.GetPodServiceMemberships(c.serviceLister, pod)

What happened: kube-controller-manager panic for What you expected to happen: no panic. How to reproduce it (as minimally and precisely as possible): No idea. I guess it's hard to reproduce. Anything else we need to know?: Found from here: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign cc Should we do something for v1.17? (v1.17 code freeze now, but still have the opportunity to fix) can be removed safely in v1.18, right?
/sig scheduling
Working on this in /sig network /priority critical-urgent /assign
Sorry, I didn't realize it. I should have a search before firing an issue. I recommend adding this issue to your PR. e.g. `
Updated, thanks!
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This seems to be writing to a cached (should be const) service. I think we need to DeepCopy() it before we mutate the Go type. does that sound right? Is this still our best state of the art?
yes, copy before change
we have a tool to catch this (the cache mutation detector), but it got disabled in our test runs two years ago (xref )
yes great catch - I was always convinced this is enabled...


Question: SyncFrequency:      *syncFrequency, HTTPCheckFrequency: *httpCheckFrequency, } <del> my_kubelet.RunKubelet(*file, *manifest_url, *etcd_servers, *address, *port) </del> <ins> my_kubelet.RunKubelet(*file, *manifestUrl, *etcdServers, *address, *port) </ins> }

It would be great to have an easy way to run a kubernetes master/minion locally. Currently, it's not that easy to set up. We should have a convenient script to make it work.
The integration test binary is pretty close. It'd be pretty simple to glue in the kubelet to that binary. Brendan On Jun 10, 2014 2:53 PM, "Daniel Smith" wrote:
I'll take a look.
+1 for this -- especially in script form rather than a go executable.
what exactly do you mean by "script form"? Were you thinking of something like a version of cloudcfg that can be run without needing to have started up a daemon first?
Basically, what I had in mind, is a shell script that looks something like
Ah, ok. That's very similar to local-; the only difference is that everything is packaged into a single go binary. We probably will switch to using individual binaries for this when we start running our components inside docker containers.
The script just makes things more obvious how to run things for those who are looking to integrate them into non-GCE environments.
That's a good point. I'll look into this some more later.
Create a pad as follows: Name: non-persistent-redis Container image: redis Named-volum with name: cache-control Mount path: /data/redis It should launch in the pre-prod namespace and the volume MUST NOT be persistent. apiVersion: v1 kind: Pod metadata: name: non-persistent-redis namespace: pre-prod spec: containers: name: non-persistent-redis image: redis volumeMounts: name: cache-control mountPath: "/data/redis" volumes: name: cache-control emptyDir: {}

I cannot find such a service when . Same thing applies for . My cluster runs fine. cc:
Ah, good catch... everything is run as a container now, will update the documentation on where to get the logs.

When I'm accessing persistentVolumes and persistentVolumeClaims, I'm getting nil instead of empty list of items, as happening in other entities when there are no items. { "kind": "PersistentVolumeList", "apiVersion": "v1", "metadata": { "selfLink": "/api/v1/persistentvolumes", "resourceVersion": "" } }
Yes, this seems like a bug. I see an empty list when I access pods: And items is nil when I access persistent volumes:
Items shouldn't be tagged with omitempty:
cc
PersistentVolumeList and PersistentVolumeClaimList appear to be the only resource types with this problem.
Please see for fix.


Question: deadline          int64 // cj status <del> ranPreviously bool stillActive   bool </del> <ins> ranPreviously              bool stillActive                bool jobPresentInCJActiveStatus bool </ins> jobCreationTime time.Time

in TestControllerV2SyncCronJob, there are many listing the properties that don't matter (like nil, false). and many constants F and T, which is unnecessary. As per 's request, We shouldn't be listing the properties that don't matter (like nil, false). And get rid of constants F and T check No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I would like to work on the optimization
It's not an optimization. It's just about readability.
/remove-kind bug /kind cleanup
/sig apps


Question: "k8s.io/kubernetes/test/integration/framework" ) <ins> func TestWebsocketWatchClientTimeout(t *testing.T) { // server setup masterConfig := framework.NewIntegrationTestControlPlaneConfig() instance, s, closeFn := framework.RunAMaster(masterConfig) defer closeFn() // object setup service := &corev1.Service{ ObjectMeta: metav1.ObjectMeta{Name: "test"}, Spec: corev1.ServiceSpec{ Ports: []corev1.ServicePort{{Name: "http", Port: 80}}, }, } configmap := &corev1.ConfigMap{ ObjectMeta: metav1.ObjectMeta{Name: "test"}, } clientset, err := kubernetes.NewForConfig(instance.GenericAPIServer.LoopbackClientConfig) if err != nil { t.Fatal(err) } if _, err := clientset.CoreV1().Services("default").Create(context.TODO(), service, metav1.CreateOptions{}); err != nil { t.Fatal(err) } if _, err := clientset.CoreV1().ConfigMaps("default").Create(context.TODO(), configmap, metav1.CreateOptions{}); err != nil { t.Fatal(err) } testcases := []struct { name         string path         string timeout      time.Duration expectResult string }{ { name:         "configmaps", path:         "/api/v1/configmaps?watch=true&timeoutSeconds=5", timeout:      10 * time.Second, expectResult: `"name":"test"`, }, { name:         "services", path:         "/api/v1/services?watch=true&timeoutSeconds=5", timeout:      10 * time.Second, expectResult: `"name":"test"`, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { u, _ := url.Parse(s.URL) apiURL := "ws://" + u.Host + tc.path wsc, err := websocket.NewConfig(apiURL, apiURL) if err != nil { log.Fatal(err) } wsConn, err := websocket.DialConfig(wsc) if err != nil { t.Fatal(err) } defer wsConn.Close() resultCh := make(chan string) go func() { defer close(resultCh) buf := &bytes.Buffer{} for { var msg []byte if err := websocket.Message.Receive(wsConn, &msg); err != nil { if err == io.EOF { resultCh <- buf.String() return } if !t.Failed() { // if we didn't already fail, treat this as an error t.Errorf("Failed to read completely from websocket %v", err) } return } if len(msg) == 0 { t.Logf("zero-length message") continue } t.Logf("Read %v %v", len(msg), string(msg)) buf.Write(msg) } }() select { case resultString := <-resultCh: if !strings.Contains(resultString, tc.expectResult) { t.Fatalf("Unexpected result:n%s", resultString) } case <-time.After(tc.timeout): t.Fatalf("hit timeout before connection closed") } }) } } </ins> func TestWatchClientTimeout(t *testing.T) { masterConfig := framework.NewIntegrationTestControlPlaneConfig() _, s, closeFn := framework.RunAMaster(masterConfig)

Using the Fabric8 client, which defaults to using WebSockets for Watches, I established a Watch on Service resources. After the timeout (implicit or explicit) rather than the WebSocket seeing a close control frame or even an EOF it continued to be open and fulfill ping/pong requests - but no Service events are sent. The WebSocket would close. Create a WebSocket based Watch with the ListOptions timeoutSeconds set on the Service resource along with a steady production of Service events, such as create/delete of a dummy Service. Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-10-27T05:15:27Z", GoVersion:"go1.15.0", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} The behavior has been seen in both local minikube and OpenShift distributions based upon Kubernetes 1.20. OS (e.g: ): Fedora 33
/sig api-machinery
Is this the same issue as ?
It could be related, but in that case the client is calling close. In this case we expect the server to initiate the close.
A couple of things were discussed when we triaged this issue: a) we need to involve the Fabric8 team (~java client?) b) We don't always default to Websocket, in fact we discussed discontinuing it c) But the code is still there, so it's allowed for now and we should support it. Not sure what to do next
a) no, it sounds like an apiserver problem, it should be reproducible with a go client in a test. b) If anything, we should standardize on websockets. Brower based clients depend on them. And when/if we get rid of SPDY, websockets are the replacement. This may be relevant to interests.

I didn't see an option to use websockets with the go client the impl is at Is there another code path or go client that will use a websocket for Watches?
Technically I didn't say our go client, just a go client. I just had in mind using the raw websockets library. must be testing the websocket watches somehow?
I adapted to It will simply keep running - with no events after the timeout. If you switch to configmap, it will terminate.
that's a terrific reproducer, thanks!
Yeah, we should absolutely be able to stick that in a unit / integration test somewhere, thanks!
opened in configmap watch works correctly, service watch hits timeout without the server closing the connection
I suspect the issue is in newDecoratedWatcher. Services implement a decorator and configmaps don't.
yup, the request context wasn't propagated to the decorated watcher
Service decorator was in 1.20 in Fix with unit and integration test in Will pick to 1.21 and 1.20 once merged
/triage accepted


Question: svc.ObjectMeta.Annotations[key] = tier } <del> func clearNetworkTier(svc *v1.Service) { key := gcecloud.NetworkTierAnnotationKey if svc.ObjectMeta.Annotations == nil { return } delete(svc.ObjectMeta.Annotations, key) } </del> // TODO: add retries if this turns out to be flaky. func reserveRegionalAddress(cloud *gcecloud.Cloud, name string, netTier cloud.NetworkTier) (string, error) { Addr := &compute.Address{

ci-kubernetes-e2e-gci-gce-coredns ci-kubernetes-e2e-gci-gce-slow ci-kubernetes-e2e-gci-gce-ipvs Basically, all jobs that run the test :) [sig-network] Services [Slow] should be able to create and tear down a standard-tier load balancer 06-05
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network it seems you reported failures in this test previously and seems a GCE feature dependent on the GCE controller, can you PTAL? cc
It has been so long that I entirely forgot what was the previous cause. From a quick look, the gce loadbalancer controller stuck on an address not found error (logs from ):
Full logs related to the problematic service : It reads like the controller thought the service desired tier network, where it actually wanted ? Would it be (purely based on the timing - don't see how that is related yet)
/assign Thanks It is indeed caused by . Pushed a PR to fix the e2e test.


Question: package cmd import ( <ins> "encoding/json" "errors" </ins> "fmt" "io" <ins> "github.com/ghodss/yaml" </ins> "github.com/spf13/cobra" <ins> apimachineryversion "k8s.io/apimachinery/pkg/version" </ins> "k8s.io/kubernetes/pkg/kubectl/cmd/templates" cmdutil "k8s.io/kubernetes/pkg/kubectl/cmd/util" "k8s.io/kubernetes/pkg/util/i18n" "k8s.io/kubernetes/pkg/version" ) <ins> type Version struct { ClientVersion *apimachineryversion.Info `json:"clientVersion,omitempty" yaml:"clientVersion,omitempty"` ServerVersion *apimachineryversion.Info `json:"serverVersion,omitempty" yaml:"serverVersion,omitempty"` } </ins> var ( version_example = templates.Examples(` # Print the client and server versions for the current context

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): "kubectl version", "kubectl version" json Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request Kubernetes version (use ): $ kubectl version --client Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-11-12T05:22:15Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): MacOS 10.12.2 I find it difficult to parse the output of programmatically. I have to use a YAML parser to do it. I'd prefer to use JSON, or to be given just the string and asked to parse it with a regex. This is obviously a relatively low-priority request.
I wouldnt mind helping out on this, if there are no objections?
are you working on this? If not, I would like to work on it.
I plan to, but if i'll keep you posted if i get too delayed.
How can I direct message you?
didnt see this till now, felt bad i had not kept up on this so i decided to just push through it. If my email is not available in my profile, please let me know, otherwise AT gmail would be the best way to contact me.
Thanks!
Included below are some examples. Let me know what you think, my only concern is the the original output of included the key and then the object type which is a little strange. Note the output flag is optional so we don't break previous behavior and contract, which is now baked and part of e2e testing. here are the couple of samples:


Question: // timeout has elapsed and then closes the channel. // // Over very short intervals you may receive no ticks before the channel is <del> // closed. A timeout of 0 is interpreted as an infinity. </del> <ins> // closed. A timeout of 0 is interpreted as an infinity, and in such a case // it would be the caller's responsibility to close the done channel. // Failure to do so would result in a leaked goroutine. </ins> // // Output ticks are not buffered. If the channel is not ready to receive an // item, the tick is skipped.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--Related to , there remains a leak in which does not ensure the channel it passes to is closed. This causes goroutines to leak under common scenarios, such as when calling . /cc What happened: Each call to leaks an active goroutine (in this case spawned by via . The goroutine will not end until the passed to is closed - however this requirement can and should be handled internally. Often callers will pass a context with a longer lifespan. What you expected to happen: should ensure that the polling goroutine is ended after its condition is met. This can be accomplished by creating a new channel with a child cancellation relationship to the original, and passing that on instead. This child channel can be closed once returns. How to reproduce it (as minimally and precisely as possible): The minimal reproduction would be to call with a true condition, and not close the passed . However, a more complete reproduction involving a can be seen . Anything else we need to know?: This leak was discovered in practice and caused a steady leak of memory and cpu in a project I work on. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Google GKE OS (e.g. from /etc/os-release): n/a Kernel (e.g. ): n/a Install tools: n/a Others: n/a <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
fyi Oren has 2 options to fix this. He'll be pasting sample code shortly and we can decide which approach we prefer.
Hi everyone! As promised, I've solved this two similar ways, and wanted to get your temperature on them before submitting the PR. I can then create a child context and pass its done channel to : This ensures that poller's goroutine will be cleaned up either when stopCh is closed or when the condition is met, whichever comes first. If there is aversion to using contexts here, we can mimic the subset of functionality from context that we need (safe to cancel from multiple goroutines + and a parent-child cancellation relationship): I lean towards option 1 and contexts - their semantics are familiar and result in less code on our end. Let me know what you think!

I would prefer using contexts - we actually should in the long term have context variants of all waits, so it’s the right direction to start using them more here
+1 to option 1
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: } ) <ins> func TestNilOutMap(t *testing.T) { var fakeKubeconfigData = `apiVersion: v1 kind: Config clusters: - cluster: certificate-authority-data: UEhPTlkK server: https://1.1.1.1 name: production contexts: - context: cluster: production user: production name: production current-context: production users: - name: production user: auth-provider: name: gcp` _, _, err := clientcmdlatest.Codec.Decode([]byte(fakeKubeconfigData), nil, nil) if err != nil { t.Fatalf("unexpected error: %v", err) } } </ins> func TestNonExistentCommandLineFile(t *testing.T) { loadingRules := ClientConfigLoadingRules{ ExplicitPath: "bogus_file",

What happened: I have a test case: This behaves different with different client-go versions. With , the test passes. This is using in with in With , this test panics. This is using in with in Panic logs: Basically it its creating an object to decode into, and this object has some maps. These are set to nil, so when we try to populate them we get a panic. What you expected to happen: Behavior remains the same as previous release, or at least have some alternative that works here? How to reproduce it (as minimally and precisely as possible): I pasted the test above that reproduces this. This can be reproduced with a project with just that file, and a file with only client-go. Using go 1.13
/sig api-machinery
/assign


Question: var _ Interface = new(Fake) <ins> func (c *Fake) Start() error { return nil } </ins> func (c *Fake) ContainerInfo(name string, req *cadvisorApi.ContainerInfoRequest) (*cadvisorApi.ContainerInfo, error) { return new(cadvisorApi.ContainerInfo), nil }

This is blocking (reboot test). In running a test that reboots all of the nodes and then runs the e2e tests, we saw a failure with the influxdb / heapster monitoring test: The heapster logs, from running (), shows a ton of the following kubelet errors for all containers (this one happens to be for skydns): The shows a bunch of 500 errors, such as: However, the apiserver thinks that all the pods are running ( gives this ) Docker thinks that the pods are running ( on that node gives this ) In summary: apiserver and docker think containers are running but kubelet can't find them this manifested in the influxdb / heapster monitoring e2e test after node reboot I've still got the cluster around if anyone would like to debug. +cc (helped me debug this (thanks!)) +cc (node) +cc (interested in getting in, which this is blocking)
Can I access your test nodes?
Is this an active problem? I'd check if the cAdvisor thinks the containers exist (it is the one returning the error). I'd then scan the logs to see if cAdvisor had any errors related to that container.
Unfortunately during debugging, I restart kubelet service on that node, the issue is gone. The problem could be cadvisor cannot get stats, but I couldn't find any cadvisor related error messages. How easy to reproduce the issue here? Next time we might access cadvisor's page to see if there is no such container information. Thanks!
(and Thanks for looking into this! I believe I can reproduce easily but I'll confirm. Where do I look for cAdvisor info? I don't see a log in or , and I don't see a cAdvisor container running?
cAdvisor is running as part of kubelet now. If you can reproduce the issue, could you please change kubelet's log level to 4 so that we have more cadvisor log here? Thanks!
Yes! How do I do that?

I can reproduce this by doing the following: After this, I got two failures: ... and I have another node with more of the same kubelet errors: (If I need to change the Kubelet log level at compile-time I can do that and repro on a new cluster; otherwise, I've got another cluster with the issue that can be introspected...)
There is a race between cadvisor initialization and docker. cadvisor failed to register docker factory, and bailed out. is going to make cadvisor more resilient to the issue of docker is not available, and have some retry logic. I am going to change kubelet initialization logic a little bit to make sure docker is running before we start cadvisor.
Awesome, thank you Dawn! Are there issues or PRs I can follow to know when this is unblocked?
is one of them. is working on the second.


Question: // podIsShutDown returns true if kubelet is done with the pod or // it was force-deleted. func podIsShutDown(pod *v1.Pod) bool { <del> // The following text is based on how pod shutdown was // initially described to me. During PR review, it was pointed out // that this is not correct: "deleteGracePeriodSeconds tells // kubelet when it can start force terminating the // containers. Volume teardown only starts after containers // are termianted. So there is an additional time period after // the grace period where volume teardown is happening." // // TODO (https://github.com/kubernetes/enhancements/issues/1698#issuecomment-655344680): // investigate what kubelet really does and if necessary, // add some other signal for "kubelet is done". For now the check // is used only for ephemeral volumes, because it // is needed to avoid the deadlock. // </del> // A pod that has a deletionTimestamp and a zero // deletionGracePeriodSeconds <del> // a) has been processed by kubelet and is ready for deletion or </del> <ins> // a) has been processed by kubelet and was set up for deletion //    by the apiserver: //    - canBeDeleted has verified that volumes were unpublished //      https://github.com/kubernetes/kubernetes/blob/5404b5a28a2114299608bab00e4292960dd864a0/pkg/kubelet/kubelet_pods.go#L980 //    - deletionGracePeriodSeconds was set via a delete //      with zero GracePeriodSeconds //      https://github.com/kubernetes/kubernetes/blob/5404b5a28a2114299608bab00e4292960dd864a0/pkg/kubelet/status/status_manager.go#L580-L592 // or </ins> // b) was force-deleted. // // It's now just waiting for garbage collection. We could wait

The implementation of generic ephemeral volumes depended on relaxing the "pod uses PVC" check such that PVC deletion is allowed to proceed even when the pod itself still exist, as long as it is certain that kubelet is done with the pod: As noted in that comment, there were diverging opinions about what deletionGracePeriodSeconds being zero means. This issue is about double-checking that and if necessary, come up with a better check.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Jordan in kubelet which causes to be set to zero: This is called "after terminating the containers and cleaning up the pod". So and are indeed equivalent. does this address the concern that you had regarding generic ephemeral volumes and pod shutdown? Whether "after terminating the containers and cleaning up the pod" is good enough for the PVC protection controller is a different question, but that then is also affecting normal PVCs and not just generic ephemeral volumes.
It looks like calls which calls : So it does look like this waits until pod volumes are successfully unmounted (minus the node unstage issue). This addresses my concern thanks!
Only thing left is to clean up the todo in the code.
See


Question: ) var ( <del> order = []string{"false", "true", "matches", "nopods", algorithmpredicates.MatchInterPodAffinityPred} </del> <ins> errPrioritize = fmt.Errorf("priority map encounters an error") order         = []string{"false", "true", "matches", "nopods", algorithmpredicates.MatchInterPodAffinityPred} </ins> ) func falsePredicate(pod *v1.Pod, meta algorithm.PredicateMetadata, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: A failed Prioritize Map Function caused Scheduler break down. What you expected to happen: Pod remains Pending and describe pod will show errors of failed functions. How to reproduce it (as minimally and precisely as possible): It happens when some prioritize's map functions return with error. Anything else we need to know?: Environment: Kubernetes version (use ): v1.9.6 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
As shown above, if priorityConfigs[i].Map(pod, meta, nodeInfo) returns with error, there will be no chance for results[i + 1] to initialize its value, thus causing panic when calling config.Reduce(pod, meta, nodeNameToInfo, results[i+1])
/sig scheduling
I opened a PR to initialize results with real host name instead of empty string.


Question: PrintFlags  *genericclioptions.PrintFlags RecordFlags *genericclioptions.RecordFlags <del> DeleteFlags   *cmddelete.DeleteFlags </del> DeleteOptions *cmddelete.DeleteOptions DryRunStrategy cmdutil.DryRunStrategy

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: ADD https://storage.googleapis.com/kubernetes-release/easy-rsa/easy-rsa.tar.gz /root/kube/ # Copy the the cni-bin folder into /opt/cni/bin <del> COPY cni-bin /opt/cni/bin </del> <ins> COPY cni-bin/bin /opt/cni/bin </ins> # Copy overlay configuration to default directory COPY cni-conf /etc/cni/net.d

We should update cni binaries to containernetworking/cni and release binaries for all arches. I do not have push access to the repo, so I guess some googler have to do it.
Thanks. I have this on my plate.


Question: format:     "{"ts":%f,"caller":"json/json_test.go:%d","msg":"test for duplicate keys","v":0,"akey":"avalue","akey":"anothervalue"}n", keysValues: []interface{}{"akey", "avalue", "akey", "anothervalue"}, }, <ins> { msg:        "test for NamespacedName argument", format:     "{"ts":%f,"caller":"json/json_test.go:%d","msg":"test for NamespacedName argument","v":0,"obj":{"Name":"kube-proxy","Namespace":"kube-system"}}n", keysValues: []interface{}{"obj", types.NamespacedName{Name: "kube-proxy", Namespace: "kube-system"}}, }, </ins> } for _, data := range testDataInfo {

should implement . This ensures that it's use as value in a key/value pair emits a struct and not the string. /ref
This depends on logr 1.2.0, which will be in /wg structure-logging /sig instrumentation
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : This depends on logr 1.2.0, which will be in /sig instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'd like to work on this :) /assign
/wg structured-logging
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /lifecycle frozen


Question: "testing" openapi_v2 "github.com/google/gnostic/openapiv2" <ins> "github.com/google/go-cmp/cmp" </ins> "k8s.io/apimachinery/pkg/api/errors" "k8s.io/apimachinery/pkg/api/meta"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When I created a crd e.g.: Then I used . I found that the first time the result will be with an error: And as for the second time it will be normal: And it can be stably reproduced every time I created a new CRD. What you expected to happen: kubectl returns the first time I requested the new CRD instead of an error. Anything else we need to know?: I also found that requesting short name e.g. will be the same. It returns an error the first time but becomes normal the second time. Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /kind bug
/area custom-resources
Confirmed that could show up even the crd has status /assign
/sig api-machinery Haven't looked into the problem yet but could be related:
Looks like this happens because cache file in .kube/http-cache/ or .kube/cache/ is not synced with kube-apiserver. delete all files under .kube/http-cache/ and .kube/cache/ will cause a sync and the error will not show up. This looks like a cli issue.
fyi
Tried to fix in cli:
/cc
/assign
this seems similar to
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is /shrug
Can you put this into words? What's the actual issue?
Even though this bug looks small, it might involve larger changes both in fixing and refactoring.. A typical corner case it is Can you put this into words? What's the actual issue? SRY, I should have put it clearer.. I just mean that this issue is sort of fundamental and obviously has been existing for a long time.
/remove-shrug
/unshrug looks like an important problem we want to solve (?)
¯()/¯ <detailsIn response to : /unshrug Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten ouch..
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
encountered same issue, as suggested the error will not show up.
any progress?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Here is a little more info, and here is how I am able to reproduce it (kubectl version 1.18.0 and cluster version 1.19.0-beta.2): A couple interesting things i noted... If I deleted my and re-ran it, I could not reproduce it. It is only the first time the short name was EVER used that this occurs. I only tried from one client machine, so I'm not sure this is cluster-specific or client-specific. If I used a different short name, like 'fo2' and re-ran the example, then it reproduces the problem. You can also run after creating the CRD and it has the same effect as deleting your . EDIT TO ADD: This doesn't seem to be a timing thing... if I sleep for 10 seconds after creating the CRD, it still has the problem.
The root of the problem is that the is run before the discovery. So in this particular case this line continues to b/c read cached data and never reached out to server. Whereas discovery mapper in will get a not found error and will proceed invalidating cache and re-fetching discovery data and return that back. In the subsequent call the shortcut expander is already aware of the new shortcut so it works just fine.

We'd need to teach shortcut expander to re-fetch discovery data, but I'm not sure that's desirable, or we could at least warn a user about this situation.
See for possible approach.
Any new progress on this ?
lists a candidate solution. This bug is causing steadily failing tests. I'll talk to a few people to see if this can get any traction. do you know who can get this moving? It sounds like you have identified the problem and have proposed a solution.


Question: go_library( name = "go_default_library", srcs = [ <ins> "capabilities.go", </ins> "doc.go", <del> "mustrunas.go", </del> "types.go", ], importpath = "k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities",

The name of this file is odd: The file context has nothing to do with Is this a BUG REPORT or FEATURE REQUEST?: /assign


Question: config := e2enetwork.NewNetworkingTestConfig(f, false, true) ginkgo.By(fmt.Sprintf("dialing(sctp) %v --> %v:%v (config.clusterIP)", config.TestContainerPod.Name, config.ClusterIP, e2enetwork.ClusterSCTPPort)) message := "hello" <del> config.DialEchoFromTestContainer("sctp", config.TestContainerPod.Status.PodIP, e2enetwork.EndpointSCTPPort, config.MaxTries, 0, message) </del> <ins> err := config.DialEchoFromTestContainer("sctp", config.TestContainerPod.Status.PodIP, e2enetwork.EndpointSCTPPort, config.MaxTries, 0, message) if err != nil { framework.Failf("failed dialing endpoint, %v", err) } </ins> }) ginkgo.It("should recreate its iptables rules if they are deleted [Disruptive]", func() {

What would you like to be : Right now we poll one node at a time for intra-pod tests. We ultimately fail a test if any one node is unreachable. If we instead polled all nodes, multiple times, and collected the number of times a node was unreachable, this would tell us what % of nodes on the cluster were broken, rather then just telling us the first node that was broken. not break the testing contract (were still verifying intra-pod connectivity, just doing so in a way that gives us holistic cluster info) Why is this needed: I described why this would make triage alot easier, and in general make it easier to see why tests are failing, here ... i'll collect a little more data on this to justify the decision .... as this is a conformance test so the bar for modification is higher (even if the test itself isnt changing) then regular tests.
/sig network
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved


Question: for _, na := range nt.tree { na.lastIndex = 0 } <del> nt.exhaustedZones = sets.NewString() </del> <ins> nt.zoneIndex = 0 </ins> } // Next returns the name of the next node. NodeTree iterates over zones and in each zone iterates

Testgrid: https://k8s- Sample failure log: https://k8s- This test job has always been flaky, but is now failing more often than it's passing. Diagnosing it is difficult, though, because even the "successful" runs have multiple failures in the log, and the reported test failures seem not to be independant failures in the logs, but rather related to multiple failures to deploy pods and storage in the test setup. Does anyone have more insight into what's going on here? This feels more like a google cloud or test config failure than anything else. /kind failing-test /kind flake /sig test-infrastructure /sig gcp /sig apps
/sig testing
also looks like https://k8s- has similar issues, so this is actually not a gke specific problem.
one of the kubelet log has not sure if that's helpful at all... /assign (redirect to get some help from dev-oncall to see if can find any leads here...)
/assign /cc
GitHub didn't allow me to assign the following users: artemvmin. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The suites are failing on both gce and gke, so I don't think it's a gke specific issue. Looking at one occurrence: https://k8s- The first test that started failing was: Because it couldn't schedule the pod. Looking at the scheduler logs, I only see these two lines referring to that pod and nothing else: This is odd because there is usually at least one more log message if scheduling failed or succeeded. This implies that the pod is stuck somewhere in the scheduler and there could potentially be a deadlock. There are no other scheduling logs about other pods after this. An odd pattern that I noticed was that for all the serial runs where a ton of tests fail, the test case that always runs before the subsequent failing tests is the equivalence cache tests. And there's always a line in the like: I wonder if there is some race condition regarding the pod deletion case.

I have seen this pattern in the most recent runs where a bunch of tests all fail: https://k8s- https://k8s- https://k8s- https://k8s- The first failing test case is different across all these runs, but the commonality between them all is that the equivalence cache tests run right before it.
can you plz help take a look at this and help triage within scheduling?
fyi, this is pretty easy to repro on a dev cluster on head. I just run the EquivalenceCache tests, and then create any Pod afterwards. The Pod fails to schedule about 50% of the time.
I am investigating it.
I identified the issue and sent a PR to address it. We need to cherrypick the fix to 1.12 as well. Thanks to for the initial investigation and finding reproduction steps!
Thanks a lot and for looking into this and giving a fox PR. /cc needs to be CP'ed into 1.12 as well.
It already got approved. Please create a cherry pick PR for it.


Question: kube::log::status "Installing godep version ${GODEP_VERSION}" go install ./vendor/github.com/tools/godep/ <ins> GP="$(echo $GOPATH | cut -f1 -d:)" hash -r # force bash to clear PATH cache PATH="${GP}/bin:${PATH}" </ins> if [[ "$(godep version 2>/dev/null)" != *"godep ${GODEP_VERSION}"* ]]; then kube::log::error "Expected godep ${GODEP_VERSION}, got $(godep version)"

Didn't see an issue for this. Godep recently updated its version to v80. This broke several builds that directly godep, grabbing latest. /kind bug cc Example https://k8s- Here are some examples where we grab master:
We have helpers to grab godep. We should use them everywhere.
/cc
/assign


Question: "regexp" "time" <ins> appsv1 "k8s.io/api/apps/v1" </ins> v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /cc /priority critical-urgent /milestone v1.19
Appears to have started failing after , which seems unrelated. /cc
according to the build log , the test timed out because it could not find gpu on any nodes. Is it probably a test-infra issue?
I think you are right, I don't see anything changing in the config so must be underlying infra issue /cc
yes, the issue is that the nodes do not have the GPU required to run the test The same infra commit () was used for the last passing job () and the first failing job ()
not sure who owns the node setup for the gpu tests, but I would guess
test-infra-maintainers owns e.g. prow, I don't think anything there relates to the GPUs the infra commit (test-infra commit) not changing suggests perhaps a change to the scripts / test harness in this repo?
the delta between the passing and failing runs only changed an ingress e2e test ()
The image is used in when installing the Nvidia driver. Given that ([1]) is lastly updated on July 17, 2020 at 12:22:11 AM UTC-7 (latest tag points to v2.0.0), does it seem this version is incompatible? BTW: the latest successful run pulled the image with SHA: 8d86a652759f80595cafed7d3dcde3dc53f57f9bc1e33b27bc3cfa7afea8d483 ([2]), maybe we can try that SHA specifically? [1]: [2]:
Forgot to mention, the shows that the deamonset was pending:
yes, and the log also shows that the pod was pending because the init container failed to complete could we change the image to  whose digest is exactly .


Question: KubeVersion:    "v1.10.0-alpha.2", KubeadmVersion: "v1.10.0-alpha.2", DNSVersion:     "1.14.7", <del> EtcdVersion:    "3.1.11", </del> <ins> EtcdVersion:    "3.1.10", </ins> }, }, },

uses non existing etcd version for k8s 1.10.0. /kind bug /sig cluster-lifecycle


Question: Preferred:        false, }, }, <ins> { name:   "Special cased PolicySingleNumaNode with one no-preference provider", policy: NewSingleNumaNodePolicy(), hp: []HintProvider{ &mockHintProvider{ map[string][]TopologyHint{ "resource1": { { NUMANodeAffinity: NewTestBitMask(0), Preferred:        true, }, { NUMANodeAffinity: NewTestBitMask(1), Preferred:        true, }, { NUMANodeAffinity: NewTestBitMask(0, 1), Preferred:        false, }, }, }, }, &mockHintProvider{ nil, }, }, expected: TopologyHint{ NUMANodeAffinity: NewTestBitMask(0), Preferred:        true, }, }, </ins> } for _, tc := range tcases {

What happened: I think I'm seeing a problem with the topology manager. I have two numa nodes, 22 cores per node. Kubelet has "--cpu-manager-policy=static --topology-manager-policy=single-numa-node ", and when I try to create a simple pod with 3 cpus and 500Mi of RAM in the guaranteed class it fails. Describing the pod gives "Resources cannot be allocated with Topology Locality", but I know there's lots of CPU and RAM available on both NUMA nodes. I'm instrumenting the topology manager kubelet code, currently there's not enough debug logs to actually be able to tell what's going wrong. What you expected to happen: I expect the pod container to be created on one or the other numa node and get allocated exclusive CPUs (via the static policy) on that NUMA node, and be affined to those CPUs. How to reproduce it (as minimally and precisely as possible): On a kubernetes node with multiple NUMA nodes, enable kubelet with static cpu manager policy and single-numa-node topology manager policy. Try to create a basic pod with a single container in the guaranteed QoS class. Anything else we need to know?: Environment: Kubernetes version (use ): sysadmin chris(keystoneadmin)]$ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"archive", BuildDate:"2019-10-09T05:41:44Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: wildcat pass hardware, 22 cores per NUMA node, two NUMA nodes. 64GB RAM per NUMA node. OS (e.g: ): CentOS 7 Kernel (e.g.

): 3.10.0-
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
oops, I thought the "close" meant the comment, not the whole issue. :)
I was able to reproduce this issue. The following (new) uint test in fails unexpectedly: With any policy other than the this test passes. This is due to 3 factors: 1) The CPUManager is the only hint provider passing actual valid hints 2) The devicemanager passes back as its hints, resulting in a single (i.e. don’t care) hint 3) We a special casing for the in the device manager that essentially turns the devicemanager's hint into a : Because the single hint from the devicemanager now has a preference, there is no chance to have it combine with any of the CPUManager hints to result in a successful hint. This should ultimately be addressed as part of the following in 1.17 But we should put a one-off patch into the 1.16.x branch to fix this in 1.16.
ptal
I've just updated with my observations. can you have a look and comment ?


Question: before the rollout finishes. release: v1.12 file: test/e2e/apps/deployment.go <ins> - testname: Deployment, completes the lifecycle of a Deployment codename: '[sig-apps] Deployment should run the lifecycle of a Deployment [Conformance]' description: When a Deployment is created it MUST succeed with the required number of replicas. It MUST succeed when the Deployment is patched. When scaling the deployment is MUST succeed. When fetching and patching the DeploymentStatus it MUST succeed. It MUST succeed when deleting the Deployment. release: v1.20 file: test/e2e/apps/deployment.go </ins> - testname: Jobs, orphan pods, re-adoption codename: '[sig-apps] Job should adopt matching orphans and release non-matching pods [Conformance]'

[x] Original approval issue: [x] Test pr: [x] Two weeks soak end date: 19 November 2020 This issue is created to allow edit by /wip /hold According to this APIsnoop query, there are still some remaining Deployment endpoints which are untested. a Deployment with a static label the Deployment with a new Label and updated data the Deployment to ensure it's patched all Deployments in all Namespaces find the Deployment(1) ensure that the Deployment is found and is patched Namespaced Deployment(1) via a Collection with a LabelSelector Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 5 points* This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
BobyMcbobs commented in original Issue: Unsure if the endpoint replaceAppsV1NamespacedDeploymentStatus is able to be hit. When this endpoint is attempted to be updated the data after retrieving the update resource doesn't reflect the values in the update
/assign
When using the appv1.Deployment structure instead of maps of interfaces to build the structure in a patch of the Deployment, this error comes up. Still in progress yment-static":"true"}: does not match template , spec.selector: Invalid value: "null": field is immutable]
replaceAppsV1NamespacedDeploymentStatus to test. Increase cover by 1 Endpoint
/reopen PR reverted by due to flaky test
Reopened this issue. <detailsIn response to : /reopen PR reverted by due to flaky test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen Soaking, ready for promotion 19 November 2020
Reopened this issue. <detailsIn response to : /reopen Soaking, ready for promotion 19 November 2020 Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle stale
/triage accepted
/merged


Question: ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", }, <ins> { Name:                     "resize-policy-cpu", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", ResizePolicy: []core.ContainerResizePolicy{ {ResourceName: "cpu", RestartPolicy: "NotRequired"}, }, }, { Name:                     "resize-policy-mem", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", ResizePolicy: []core.ContainerResizePolicy{ {ResourceName: "memory", RestartPolicy: "RestartContainer"}, }, }, { Name:                     "resize-policy-cpu-and-mem", Image:                    "image", ImagePullPolicy:          "IfNotPresent", TerminationMessagePolicy: "File", ResizePolicy: []core.ContainerResizePolicy{ {ResourceName: "memory", RestartPolicy: "NotRequired"}, {ResourceName: "cpu", RestartPolicy: "RestartContainer"}, }, }, </ins> } if errs := validateContainers(successCase, volumeDevices, nil, field.NewPath("field"), PodValidationOptions{}); len(errs) != 0 { t.Errorf("expected success: %v", errs)

The newly field field is not being validated. field should be validated. We can create a pod with invalid to reproduce this. No response_ <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /sig node /priority important-soon /milestone v1.27 adding to 1.27 milestone to resolve if possible
I'll try this for v1.27 if you don't mind.
getting a PR open, reviewed, and green is always good. we'll evaluate it for risk and if there's a second release candidate we can likely include it
Nice catch! Please review fix: This likely got dropped either during port-over of the or during bad rebase. I should not have left the . Gotta run to get my kid prepped, dr. appt and then school drop. So I'll try and add an E2E test for this later today. please hold for e2e test (but we should fix this in 1.27)
I realized you were working on a fix as well only after I sent my fix out and was catching up on emails at doc office (rushed morning). I didn't mean to step over your effort to fix it, sorry about that.
I am completely fine with that. Thanks for the fix :)


Question: claimSize:    "1Gi", expectedSize: "1Gi", nodeName:     node.Name, <ins> attach:       true, </ins> } }

brought this up during the CSI Standup. It seems to be true, in the test config we do not set which is required to bring up the pod for consuming the volume. The code where is checked: The test config used where is not set to true: /kind bug /sig storage /assign /cc


Question: rs.Labels["name"] = rsName initialDelaySeconds := framework.PodStartTimeout.Seconds() + 30 if tc.podsShouldBecomeReadyFirst { <del> initialDelaySeconds = 0 </del> <ins> initialDelaySeconds = 20 </ins> } rs.Spec.Template.Spec.Containers[0].ReadinessProbe = &v1.Probe{ ProbeHandler: v1.ProbeHandler{

master-blocking: gce-ubuntu-master-containerd No response /sig apps
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign


Question: <ins> # Collecting log files from within containers with Fluentd and sending to the Google Cloud Logging service. This directory contains the source files needed to make a Docker image that collects log files from arbitrary files within a container using [Fluentd](http://www.fluentd.org/) and sends them to GCP. The image is designed to be used as a sidecar container as part of a pod. It lives in the Google Container Registry under the name `gcr.io/google_containers/fluentd-sidecar-gcp`. This shouldn't be necessary if your container writes its logs to stdout or stderr, since the Kubernetes cluster's default logging infrastructure will collect that automatically, but this is useful if your application logs to a specific file in its filesystem and can't easily be changed. In order to make this work, you have to add a few things to your pod config: 1. A second container, using the `gcr.io/google_containers/fluentd-sidecar-gcp:1.0` image to send the logs to Google Cloud Logging. 2. A volume for the two containers to share. The emptyDir volume type is a good choice for this because we only want the volume to exist for the lifetime of the pod. 3. Mount paths for the volume in each container.  In your primary container, this should be the path that the applications log files are written to. In the secondary container, this can be just about anything, so we put it under /mnt/log to keep it out of the way of the rest of the filesystem. 4. The `FILES_TO_COLLECT` environment variable in the sidecar container, telling it which files to collect logs from. These paths should always be in the mounted volume. To try it out, make sure that your cluster was set up to log to Google Cloud Logging when it was created (i.e. you set `LOGGING_DESTINATION=gcp` or are running on Container Engine), then simply run ``` kubectl create -f logging-sidecar-pod.yaml ``` You should see the logs show up in the log viewer of the Google Developer Console shortly after creating the pod. To clean up after yourself, simply run ``` kubectl delete -f logging-sidecar-pod.yaml ``` </ins>

We should provide a dockerfile for a pre-configured Fluentd image that can be passed file paths to collect logs from. We'll likely want one for exporting to elasticsearch and one for exporting to Google Cloud Logging, as we have today for the per-node stdout/stderr collectors. How to collect log files from within a container is a question that comes up occasionally (including ), and it provides a strong example of the usefulness of sidecar containers. cc
A controller that keeps one instance of a pod alive on each node would be useful for this. Shall I go ahead and write such a controller?
Would it be? This is intended to run as a sidecar container in a pod with the user's actual application. I think a pod-per-node controller is orthogonal (and is being discussed in and ). I'll create the actual Dockerfile, and a quick example of how to use it.
Sorry, my rush to catch up I think I mis-understood this thread.


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/sets" <ins> "k8s.io/client-go/kubernetes/fake" </ins> cloudvolume "k8s.io/cloud-provider/volume" )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have pretty large clusters in aws and in production, we found an inefficiency in provisioning dynamic EBS volumes, where aws-ebs volume provisioner tries to look for available zone from aws directly. With scale, this can trigger massive from ec2, and affect cluster stability. This is also related to What you expected to happen: To simply get zones with available instances, we can get information from API server for efficiency. There might be a delay between node gets shutting down and such change is reflected in api server, but in case node is not ready or gets terminated, there will be higher level reconciliation triggered so we are safe. How to reproduce it (as minimally and precisely as possible): We only expect this to be hit when the cluster is huge (or aws account is huge) so not easy to reproduce. /assign /sig aws /sig cloud-provider
/assign


Question: <del> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cdi import ( "testing" "github.com/stretchr/testify/assert" "k8s.io/kubernetes/pkg/kubelet/container" ) func TestGenerateAnnotations(t *testing.T) { testCases := []struct { description         string deviceIDs           []string expecteError        error expectedAnnotations []container.Annotation }{ { description: "no devices", deviceIDs:   []string{}, }, { description:         "one device", deviceIDs:           []string{"vendor.com/class=device1"}, expectedAnnotations: []container.Annotation{{Name: "cdi.k8s.io/test-driver-name_test-claim-uid", Value: "vendor.com/class=device1"}}, }, } as := assert.New(t) for _, tc := range testCases { t.Run(tc.description, func(t *testing.T) { annotations, err := GenerateAnnotations("test-claim-uid", "test-driver-name", tc.deviceIDs) as.ErrorIs(err, tc.expecteError) as.Equal(tc.expectedAnnotations, annotations) }) } } </del>

Kubelet should stop using annotations to pass CDI device IDs to CRI runtimes. CDIDevices CRI field should be used for this purpose. This should be done when Kubelet that doesn't support CRI field reaches EOL and two major CRI runtimes support CRI field. Here is a PR comment that gives more details about this: Kubelet started using CDIDevices CRI field in . Previous version 1.27 will reach EOL .CRI-O started using it in . Previous version 1.27 will reach EOL together with Kubernetes 1.27.Containerd started using it in . Version 1.7 will reach EOL in a 6 months after Containerd 2.0 release Previous version 1.6 doesn't support CDI injections and should not be considered. So, safer option would be to wait until Containerd 1.7 reaches EOL. However it's relatively safe to do it earlier when 1.7.0 and 1.7.1 have little chances to be in real use. Containerd v1.7.0 and v1.7.1 is not used by the Linux distributions from , so it's safe to remove annotations support from the Kubelet in 1.31 in my opinion.
/sig node
/triage accepted /priority important-longterm
/assign


Question: <del> v1.30.0-go1.22-bullseye.0 </del> <ins> v1.30.0-go1.22.0-bullseye.0 </ins>

https://prow.ppc64le- Tests are not getting executed due to improper go version. It has been failing since past 4 days. No response No response No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This seems to be failing due to improper version addition in dependency files where is instead of at some places. I will raise a PR for it.
/assign
fixing that


Question: overrides      *ConfigOverrides fallbackReader io.Reader configAccess   ConfigAccess <ins> // promptedCredentials store the credentials input by the user promptedCredentials promptedCredentials </ins> } // NewDefaultClientConfig creates a DirectClientConfig using the config.CurrentContext as the context name func NewDefaultClientConfig(config clientcmdapi.Config, overrides *ConfigOverrides) ClientConfig { <del> return &DirectClientConfig{config, config.CurrentContext, overrides, nil, NewDefaultClientConfigLoadingRules()} </del> <ins> return &DirectClientConfig{config, config.CurrentContext, overrides, nil, NewDefaultClientConfigLoadingRules(), promptedCredentials{}} </ins> } // NewNonInteractiveClientConfig creates a DirectClientConfig using the passed context name and does not have a fallback reader for auth information func NewNonInteractiveClientConfig(config clientcmdapi.Config, contextName string, overrides *ConfigOverrides, configAccess ConfigAccess) ClientConfig { <del> return &DirectClientConfig{config, contextName, overrides, nil, configAccess} </del> <ins> return &DirectClientConfig{config, contextName, overrides, nil, configAccess, promptedCredentials{}} </ins> } // NewInteractiveClientConfig creates a DirectClientConfig using the passed context name and a reader in case auth information is not provided via files or flags func NewInteractiveClientConfig(config clientcmdapi.Config, contextName string, overrides *ConfigOverrides, fallbackReader io.Reader, configAccess ConfigAccess) ClientConfig { <del> return &DirectClientConfig{config, contextName, overrides, fallbackReader, configAccess} </del> <ins> return &DirectClientConfig{config, contextName, overrides, fallbackReader, configAccess, promptedCredentials{}} </ins> } func (config *DirectClientConfig) RawConfig() (clientcmdapi.Config, error) {

When I want to use basic-auth, kubectl will display the password I entered: I think it is more sensible to hide the password?
And it seems that kubectl request for username/password twice...
I'll take this. There is a library here: That appears to do the trick, but we need some refactoring so that it can read from an arbitrary reader to fit our usecase.
This was fixed by but I didn't close this issue. Closing it now.


Question: package v1alpha1 import ( <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema"

Is this a BUG REPORT or FEATURE REQUEST?: Bug report /kind bug What happened: I'm following the directions to run the sample controller on When running it as directed, it gives these errors: E1214 10:34:25. ] Failed to list v1alpha1.Foo: v1.ListOptions is not suitable for converting to "" E1214 10:34:26. ] Failed to list v1alpha1.Foo: v1.ListOptions is not suitable for converting to "" What you expected to happen: Not to generate these errors. How to reproduce it (as minimally and precisely as possible): 1) Have a working kube config. 2) checkout kubernetes/sample-controller 3) run it: Anything else we need to know?: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.6", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-09-14T06:55:55Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9+", GitVersion:"v1.9.0-alpha.0.845+-dirty", GitCommit:"", GitTreeState:"dirty", BuildDate:"2017-10-02T22:37:04Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): my workstation is 14.04.5 LTS, Trusty Tahr Kernel (e.g. ): my workstation kernel is 4.4.0-96-generic Install tools: Others: go version 1.9
cc - Hi, it looks like you checked this in with . What all do I need to do to get this to work?
/sig api-machinery
Hey - thanks for this report. I'm not sure how, but we were missing an extra line in I've put in to resolve this (and it's working for me now :tada:)


Question: <ins> /* Copyright 2016 The Kubernetes Authors All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package types // int64 is used as a safe bet against wrap-around (uid's are general // int32) and to support uid_t -1, and -2. type UnixUserID int64 type UnixGroupID int64 </ins>

We have a couple fields in the API for Unix UID and GID. We should define and and use them everywhere.
Is really correct? As you pointed out, GID is an unsigned int32, isn't it?
If I recall, POSIX does not actually specify signedness and some Unixes differ on it. int64 seems like a safe superset, unless we know of any OS that uses 64 bit UID/GID ? On Oct 16, 2015 6:58 AM, "Paul Morie" wrote:
I personally don't know of one. How about you ?
int64 also has the nice property of allowing negative values without wraparound. (uidt)-1 is a thing, sadly, and (uidt)-2 is apparently a thing, too. , Paul Morie wrote:
Okay, you've got me on the fence.
I will change and will change when he's back from PTO.
We might want to talk through the effects of changing this on the origin policy stuff before we start doing it.
Okay, after talking to I suggest that we do the following: Change the types of and to and respectively in and Have another commit that makes the following type changes: Introduce and as has suggested in this issue Changes the types in and in applicable fields This will be easier for us to consume in our rebase of OpenShift onto Kubernetes than if we modify and before going into master. Any objections?
I don'[t object as long as the final goal is correct , Paul Morie wrote:


Question: width := widthEstimator.EstimateWidth(r) digest := utilflowcontrol.RequestDigest{RequestInfo: requestInfo, User: user, Width: width} <del> fcIfc.Handle(ctx, digest, note, func(inQueue bool) { if inQueue { noteWaitingDelta(1) } else { noteWaitingDelta(-1) </del> <ins> if isWatchRequest { // This channel blocks calling handler.ServeHTTP() until closed, and is closed inside execute(). // If APF rejects the request, it is never closed. shouldStartWatchCh := make(chan struct{}) watchInitializationSignal := newInitializationSignal() // This wraps the request passed to handler.ServeHTTP(), // setting a context that plumbs watchInitializationSignal to storage var watchReq *http.Request // This is set inside execute(), prior to closing shouldStartWatchCh. // If the request is rejected by APF it is left nil. var forgetWatch utilflowcontrol.ForgetWatchFunc defer func() { // Protect from the situation when request will not reach storage layer // and the initialization signal will not be send. if watchInitializationSignal != nil { watchInitializationSignal.Signal() } // Forget the watcher if it was registered. // // // This is race-free because by this point, one of the following occurred: // case <-shouldStartWatchCh: execute() completed the assignment to forgetWatch // case <-resultCh: Handle() completed, and Handle() does not return //   while execute() is running if forgetWatch != nil { forgetWatch() } }() execute := func() { noteExecutingDelta(1) defer noteExecutingDelta(-1) served = true setResponseHeaders(classification, w) forgetWatch = fcIfc.RegisterWatch(requestInfo) // Notify the main thread that we're ready to start the watch. close(shouldStartWatchCh) // Wait until the request is finished from the APF point of view // (which is when its initialization is done). watchInitializationSignal.Wait() </ins> } <del> }, execute) </del> <ins> // Ensure that an item can be put to resultCh asynchronously. resultCh := make(chan interface{}, 1) // Call Handle in a separate goroutine. // The reason for it is that from APF point of view, the request processing // finishes as soon as watch is initialized (which is generally orders of // magnitude faster then the watch request itself). This means that Handle() // call finishes much faster and for performance reasons we want to reduce // the number of running goroutines - so we run the shorter thing in a // dedicated goroutine and the actual watch handler in the main one. go func() { defer func() { err := recover() // do not wrap the sentinel ErrAbortHandler panic value if err != nil && err != http.ErrAbortHandler { // Same as stdlib http server code. Manually allocate stack // trace buffer size to prevent excessively large logs const size = 64 << 10 buf := make([]byte, size) buf = buf[:runtime.Stack(buf, false)] err = fmt.Sprintf("%vn%s", err, buf) } // Ensure that the result is put into resultCh independently of the panic. resultCh <- err }() // We create handleCtx with explicit cancelation function. // The reason for it is that Handle() underneath may start additional goroutine // that is blocked on context cancellation. However, from APF point of view, // we don't want to wait until the whole watch request is processed (which is // when it context is actually cancelled) - we want to unblock the goroutine as // soon as the request is processed from the APF point of view. // // Note that we explicitly do NOT call the actuall handler using that context // to avoid cancelling request too early. handleCtx, handleCtxCancel := context.WithCancel(ctx) defer handleCtxCancel() // Note that Handle will return irrespective of whether the request // executes or is rejected. In the latter case, the function will return // without calling the passed `execute` function. fcIfc.Handle(handleCtx, digest, note, queueNote, execute) }() select { case <-shouldStartWatchCh: watchCtx := utilflowcontrol.WithInitializationSignal(ctx, watchInitializationSignal) watchReq = r.WithContext(watchCtx) handler.ServeHTTP(w, watchReq) // Protect from the situation when request will not reach storage layer // and the initialization signal will not be send. // It has to happen before waiting on the resultCh below. watchInitializationSignal.Signal() // TODO: Consider finishing the request as soon as Handle call panics. if err := <-resultCh; err != nil { panic(err) } case err := <-resultCh: if err != nil { panic(err) } } } else { execute := func() { noteExecutingDelta(1) defer noteExecutingDelta(-1) served = true setResponseHeaders(classification, w) handler.ServeHTTP(w, r) } fcIfc.Handle(ctx, digest, note, queueNote, execute) } </ins> if !served { setResponseHeaders(classification, w)

As visible in this graph: ! pod-startup time in 5k-node tests increased around beginning of June (only 99% though). It seems to be connected to "run-to-watch" phase. We should investigate and see what we can do with this. /sig scalability /kind bug
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : As visible in this graph: ! /kind bugs Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted Once PR that merged around that time is P&F support for watch-initialitation: As a first step, we may want to confirm/reject if it's this PR and see if we aren't somehow misconfigured here.
I've briefly checked test-infra/perf-tests to see if we had any relevant changes around that time, but I don't see any. Most likely it's related to k/k changes.
/assign
Good news is that we can also see it regression in kubemark (although not as bad as in regular 5k tests): http://perf- But this can help us with finding root cause faster/cheaper. ! I've picked two runs, one before this change, one after: 26th May: Timeline: 12th June: Timeline: First run vs second run I can see significantly higher spike on waiting time in P&F, but according to timeline it seems like it was during huge service and not pod startup latency: 26th May: ! 12th June: ! I can see that number of apiserver goroutines significantly increased (from ~500k to ~900k). I would expect to have 1 more goroutine per watch, we have around 200k watches). ! vs ! Worst 100 pods with slowest latency runtowatch: I've picked the slowest pod "test-2xppkn-33/latency-deployment-60--p7nwz" to analyse.

This pod was bound to node gce-scale-cluster-minion-group-4-txww: Kubelet log for this pod: CL2 logs for this pod: In apiserver logs we can see: So it looks like updating pod status was delayed by P&F by ~500 ms for this pod. Timeline for this pod: 18:37:46. scheduled 18:37:48. container started 18:37:49. kubelet gets pod (1.3s latency with 610 ms delay from P&F) 18:37:50. patch status of pod (545ms latency with 500ms delay from P&F) 18:37:51. observed by CL2 as running To compare with previous run we have this timeline for pod "test-rcdjgd-26/latency-deployment-69--6s95x 2.724031741s": 18:57:08. scheduled on node gce-scale-cluster-minion-group-3-l21s 18:57:10. container started 18:57:10. kubelet gets pod (364 ms latency with 345 ms delay from P&F) 18:57:11. patched status od pod (360 ms latency with 254 ms delay from P&F) So to compare, for these two pods major difference was "get pod" + "patch status" in kubelet: 1.8s vs 700ms. 1s of 1.8s of latency was due to P&F and 0.6s of 0.7s was due to P&F.
In both runs I don't see significant waiting time (99th percentile) on any of Priority levels during pod startup latency phase: 26th May: ! 12th June: !
1) Do we have data about 100% waiting time in P&F? With 99% it my not be enough to to see that data. 2) If really waiting time is small, then the question is what exactly. (a) it might be related to the number of goroutines (b) it might be locking (c) it might be something else... I have just opened the WIP PR: that should help with first two (at least to some extent). Will you be able to test if that helps?

Max waiting time in P&F for pod startup latency phase: 26th May: ! 12th June: ! Buckets are ..., 0.2, 0.5, ... so I don't think it's significant. will test it.
This is definitely non-negligible regression that we should address in 1.22 timeframe. I'm actually have a promissing results for it. /milestone v1.22 /priority critical-urgent
OK - so we proved that addresses the problem: it reduces back to previous value the number of goroutines ! it brings back pod startup time back to less than 3.4s on 99th percentile it brings back API call latencies I'm going to work with reviewers on merging this to fix the regression. - FYI


Question: klog.Infof(log("blockMapper.stageVolumeForBlock STAGE_UNSTAGE_VOLUME capability not set. Skipping MountDevice...")) return "", nil } <del> publishVolumeInfo := attachment.Status.AttachmentMetadata </del> <ins> publishVolumeInfo := map[string]string{} if attachment != nil { publishVolumeInfo = attachment.Status.AttachmentMetadata } </ins> nodeStageSecrets := map[string]string{} if csiSource.NodeStageSecretRef != nil { nodeStageSecrets, err = getCredentialsFromSecret(m.k8s, csiSource.NodeStageSecretRef)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: During I develop a driver that supports raw block devices but do not need to be attached to nodes, I found it would fail when a pod mount a raw block device. PS: I use cluster-driver register and make requiredAttach as false. kubectl get csidriver csi-xxx -o yaml ! What you expected to happen: Mount a raw block device successfully. How to reproduce it (as minimally and precisely as possible): driver that supports raw block devices but do not need to be attached to nodes, I found it would fail when a pod mount a raw block device. cluster-driver register and make requiredAttach as false. a pod with raw block devices Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 Cloud provider or hardware configuration: BaiduCloud OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig storage /assign
I would send a PR later


Question: if err == nil { t.Errorf("expected error too old") } <ins> if _, ok := err.(*errors.StatusError); !ok { t.Errorf("expected error to be of type StatusError") } </ins> } { result, err := store.GetAllEventsSince(2)

We sorta fixed this-- now it's a real status error, but we left the status code 500, so it still produces stack traces and potentially confuses clients. we should figure out the correct status code.
I think there is a lot more going on related to this in . Clients are re-listing after the error but not rewatching.
/sub
It turned out this was not causing job e2e flakes. Just sending 500s that were confusing. So lowering priority.
How about a NewBadRequest error?
that certainly sounds like an improvement.


Question: kubectl create -f $KUBE_REPO/examples/mysql-wordpress-pd/local-volumes.yaml ``` ### GCE Persistent Disk This storage option is applicable if you are running on

Version: Running this example on RHEL 7.2 (assume same for centos), and using hostPath, I get a permissions error when the mysql container does a . If I set selinux to permissive on the host it works. If I manually run the container with it works. I was not able to get it to work when adding to the deployment. My fix was to locate the PVs in by changing the local- file, and doing . I this comment to local- If this issue is valid I will create a quick PR to doc this selinux solution.
See pr


Question: UserAssignedIdentityID string `json:"userAssignedIdentityID,omitempty" yaml:"userAssignedIdentityID,omitempty"` // The ID of the Azure Subscription that the cluster is deployed in SubscriptionID string `json:"subscriptionId,omitempty" yaml:"subscriptionId,omitempty"` <ins> // Identity system value for the deployment. This gets populate for Azure Stack case. IdentitySystem string `json:"identitySystem,omitempty" yaml:"identitySystem,omitempty"` </ins> } // GetServicePrincipalToken creates a new service principal token based on the configuration func GetServicePrincipalToken(config *AzureAuthConfig, env *azure.Environment) (*adal.ServicePrincipalToken, error) { <ins> var tenantID string if strings.EqualFold(config.IdentitySystem, ADFSIdentitySystem) { tenantID = "adfs" } else { tenantID = config.TenantID } </ins> if config.UseManagedIdentityExtension { klog.V(2).Infoln("azure: using managed identity extension to retrieve access token") msiEndpoint, err := adal.GetMSIVMEndpoint()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : We need to enhance Azure cloud provider code to support both AAD and ADFS authentication. Why is this needed: On Azure Stack customer can have two type of deployment .
/sig azure
/assign


Question: // ListAndWatch lists devices and update that list according to the Update call func (m *Stub) ListAndWatch(e *pluginapi.Empty, s pluginapi.DevicePlugin_ListAndWatchServer) error { log.Println("ListAndWatch") <del> var devs []*pluginapi.Device for _, d := range m.devs { devs = append(devs, &pluginapi.Device{ ID:     d.ID, Health: pluginapi.Healthy, }) } s.Send(&pluginapi.ListAndWatchResponse{Devices: devs}) </del> <ins> s.Send(&pluginapi.ListAndWatchResponse{Devices: m.devs}) </ins> for { select {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: For now, UT in doesn't work because callback function will never be called, is rewritten in this function : Another thing, i'm wondering why we rewrite device health in function which in this file: What you expected to happen: UT assertion should be executed. How to reproduce it (as minimally and precisely as possible): Add some logs in callback function of , it will show nothing. Anything else we need to know?: I will send a PR to fix this. /cc WDYT? /sig node
/assign
GitHub didn't allow me to assign the following users: RenaudWasTaken. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Good catch


Question: } for _, test := range tests { <del> equal, err := equalIgnoreHash(test.former, test.latter) if err != nil { t.Errorf("In test case %q, returned unexpected error %v", test.test, err) } if equal != test.expected { t.Errorf("In test case %q, expected %v", test.test, test.expected) } equal, err = equalIgnoreHash(test.latter, test.former) if err != nil { t.Errorf("In test case %q (reverse order), returned unexpected error %v", test.test, err) } if equal != test.expected { t.Errorf("In test case %q (reverse order), expected %v", test.test, test.expected) </del> <ins> runTest := func(t1, t2 api.PodTemplateSpec, reversed bool) { // Set up t1Copy, err := api.Scheme.DeepCopy(t1) if err != nil { t.Errorf("Failed setting up the test: %v", err) } t2Copy, err := api.Scheme.DeepCopy(t2) if err != nil { t.Errorf("Failed setting up the test: %v", err) } reverseString := "" if reversed { reverseString = " (reverse order)" } // Run equal, err := equalIgnoreHash(t1, t2) // Check if err != nil { t.Errorf("In test case %q%s, expected no error, returned %v", test.test, reverseString, err) } if equal != test.expected { t.Errorf("In test case %q%s, expected %v", test.test, reverseString, test.expected) } if t1.Labels == nil || t2.Labels == nil { t.Errorf("In test case %q%s, unexpected labels becomes nil", test.test, reverseString) } if !reflect.DeepEqual(t1, t1Copy) || !reflect.DeepEqual(t2, t2Copy) { t.Errorf("In test case %q%s, unexpected input template modified", test.test, reverseString) } </ins> } <ins> runTest(test.former, test.latter, false) // Test the same case in reverse order runTest(test.latter, test.former, true) </ins> } }

Context: NodeSwap feature gate was recently . Currently, as per , swap is only supported with cgroup v2 - cgroup v1 is no longer supported. For this reason, we decided to fail kubelet if NodeSwap feature gate is enabled and the node runs with cgroup v1. This would help prevent any surprises for the user by failing as soon as possible. This effort is being implemented in this PR: The problem: Currently, swap jobs are running with cgroup v1. Swap tests were in a way that they first check on which environment they're being run (e.g. which cgroup version? is NodeSwap FG enabled? Which swapBehavior is defined? etc) and then expect the right behavior to take place. With v1, the tests expect the containers to be defined without swap access. While the tests pass, they are much less valuable, since all they check is that swap is disabled for v1 workloads. The tests would be much more valuable if they would run with v2, therefore actually ensure that proper swap limitations are configured. In addition, is dependent on these changes, since as of now the jobs immediately fail with this PR since kubelet fails on an environment with cgroup v1 and swap enabled. What needs to change: [x] CRI-O swap related jobs should be defined to run with cgroup v2 (e.g. ) [x] containerd swap related jobs should be defined to run with cgroup v2 (e.g. ). Future work: In the future, we might consider introducing more jobs to check swap. Ideally, both and would be tested, perhaps by periodic jobs.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/sig testing
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details