Question: package network import ( <ins> "context" </ins> "fmt" "strings" "time"

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: n/a Which test(s) are failing: "should resolve connection reset issue [Slow]" Since when has it been failing: it never worked Testgrid link: n/a Reason for failure: The image used for the test is using only IPv4 Anything else we need to know: We should: [x] refactor the code to support IPv6 [ ] ~consolidate the image in agnhost~ [x] configure an IPv6 job to test network https://k8s-,%20IPv6,%20master
/assign /sig-network
GitHub didn't allow me to assign the following users: Billy99. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /sig-network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Those labels are not set on the issue: <detailsIn response to : /remove-triage unresolved Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help wanted
/help wanted /good-first-issue
/remove-good-first-issue the agnhost promotion process is not something easy for new contributors
Hi, I would like to contribute to this.
good, thanks
sorry , I needed to fix it and I took over, thanks for stepping up /assign /remove help-wanted
OK.Yesterday I have refactor the code to support IPv6, but I understand wrong about consolidate the image in agnhost. I'm sorry for not being able to help.
no worries, it was trickier than I thought, thanks again for your help
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: return grpc.Dial( string(addr), <ins> grpc.WithAuthority("localhost"), </ins> grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithContextDialer(func(ctx context.Context, target string) (net.Conn, error) { return (&net.Dialer{}).DialContext(ctx, network, target)

When registering a new driver the client is sending the filepath of the socket as the / header instead of . In practice almost all / servers outright reject this as invalid. Given that the client was updated years ago I'm unclear why this problem still persists to this day :( kubelet logs look like this: csi server logs look like this: Registration to move along happily. simple instructions here: Most drivers are written in and the server simply ignores this non-compliant header, otherwise the issue would be far more prevalent. <detailsEvery version up until current AFAIK. </details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
I ran into this error in my use-case and debugged what's going on. I think the problem goes away with a recent enough and when the socket path is correctly prefixed in the client connection: I though I could start working on fixing this in kubelet.
That would be awesome! Then I can finally do away with all the crazy workarounds.
The nomad issue link you gave tried the approach I was talking about without luck it seems. I need take a closer look what's wrong...
btw, this look a duplicate of and
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale currently planning to use but I have some cleanup pending with first
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale The fix PR is pending approval

When using the CSI driver node registration sidecar container, the kubelet-registration-path parameter is set to either or . One or both of these options should cause kubelet to send a valid authority header to the socket. In the former case, kubelet will fail to find the file path since it will pass the unix header in the net.Dialer target to dial. In the latter, the dialer will succeed to call the container over the socket but send an incorrect :authority pseudo header (the ). A call into the CSI container with a valid authority header, using one or either kubelet-registration-path parameter. Deploy a CSI plugin example, setting kubelet-registration-path on the node-driver-registrar sidecar container to either a or . Note, a plugin may not fail in the latter case if the CSI plugin is written in a language with an http2 library that does not strictly check the authority header, however the header will still be incorrect. If using Rust as a language, the h2 library will strictly check the authority header and return a protocol error. Go does not seem to reject the invalid authority header, which is perhaps why most plugins do not notice the issue. Using will not have a 'unix:' header. It checks for this to substitute 'localhost' as the authority , which will not happen in this case. Using will get the authority substituted, but will pass the full 'unix:' header in as part of the path file. Related code can be seen (non-nil custom dialer) and (newGrpcConn looks like it expects no unix header based on log entry and externally supplied dialcontext). This work may have been overlapping with related work in grpc-go and plans , where both libraries seem to be taking responsibility for managing the authority header for unix sockets now. Kubelet logs using the unix header: Of note: the inconsistent 'Error while dialing dial unix' entries: one showing the unix: header for the path, while the registration socket shows it without. Also, the 'localhost' replacement is visible in the logs for the first (unix:-prefixed) and not the second registration notification call (non-prefixed). node registration sidecar logs when not_ using the unix header: Of note, kubelet notifies the container it received a protocol error.

The CSI container rust logs show the matching protocol error and authority header value: <details</details<detailsBare metal </details<details</details<details</details<detailscri-o </details<detailscalico, metal-lb, bgp, applicable 1.23 versions. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

ref: Target should start with unix: to avoid setting empty authority header according to RFC-3986.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
looks same as and
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue.


Question: --image "${MASTER_IMAGE}"  --tags "${MASTER_TAG}"  --network "${NETWORK}"  <del> --scopes "storage-ro" "compute-rw"  </del> <ins> --scopes "storage-ro,compute-rw"  </ins> --can-ip-forward  --metadata-from-file  <del> "startup-script=${KUBE_ROOT}/cluster/gce/configure-vm.sh"  "kube-env=${KUBE_TEMP}/master-kube-env.yaml"  --disk name="${MASTER_NAME}-pd" device-name=master-pd mode=rw boot=no auto-delete=no </del> <ins> "startup-script=${KUBE_ROOT}/cluster/gce/configure-vm.sh,kube-env=${KUBE_TEMP}/master-kube-env.yaml"  --disk "name=${MASTER_NAME}-pd,device-name=master-pd,mode=rw,boot=no,auto-delete=no" </ins> } # TODO(dawnchen): Check $CONTAINER_RUNTIME to decide which

This happens over and over. Scary messages. WARNING: Lists should be separated by commas, try "--allow=tcp,udp,icmp,esp,ah,sctp". If you intend to use udp, icmp, esp, ah, sctp for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--scopes=storage-ro,compute-rw". If you intend to use compute-rw for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--metadata-from-file=startup-script=/usr/local/google/home/thockin/src/kubernetes/hack/e2e-internal/../../cluster/../cluster/gce/../../cluster/../cluster/gce/configure-,kube-env=/tmp/kubernetes.MURmzX/master-kube-". If you intend to use kube-env=/tmp/kubernetes.MURmzX/master-kube- for positional arguments, put them before the flag. WARNING: Lists should be separated by commas, try "--disk=name=e2e-test-thockin-master-pd,device-name=master-pd,mode=rw,boot=no,auto-delete=no". If you intend to use device-name=master-pd, mode=rw, boot=no, auto-delete=no for positional arguments, put them before the flag.
That's a new one. gcloud started getting a little more anal here. Exciting. cc
They changed list argument syntax a while back, but the warnings are new. Time to update startup scripts I guess.


Question: if err != nil { return err } <ins> type unidentifiedContainer struct { // Docker ID. id string // Docker container name name string } unidentifiedContainers := make([]unidentifiedContainer, 0) </ins> uidToIDMap := map[string][]string{} for _, container := range containers { _, uid, name, _ := dockertools.ParseDockerName(container.Names[0]) <ins> if uid == "" && name == "" { unidentifiedContainers = append(unidentifiedContainers, unidentifiedContainer{ id:   container.ID, name: container.Names[0], }) continue } </ins> uidName := string(uid) + "." + name uidToIDMap[uidName] = append(uidToIDMap[uidName], container.ID) } <ins> // Remove all non-running unidentified containers. for _, container := range unidentifiedContainers { data, err := kl.dockerClient.InspectContainer(container.id) if err != nil { return err } if data.State.Running { continue } glog.Infof("Removing unidentified dead container %q with ID %q", container.name, container.id) err = kl.dockerClient.RemoveContainer(docker.RemoveContainerOptions{ID: container.id}) if err != nil { return err } } // Evict dead containers according to our policies. </ins> for _, list := range uidToIDMap { if len(list) <= kl.maxContainerCount { continue

W0312 15:24:56. 792 ] found a container with the "k8s" prefix, but too few fields (5): "k8scadvisor.8d424740cadvisor-" These are all brand new pods - they should not be invalid
I've been tracking this issue, this seems to be old, not new containers. Typically happens when a GCE cluster is started (static pods are brought up by the old Kubelet) when the new Kubelet comes up it kills the old pods but we can't GC them because we can't recognize them. The log is from GC. Assigning to myself and I'll fix today.
dup of ?
Thanks closed that in favor of this. , Dawn Chen wrote:
thanks for handling this. Let me know if there's anything I can help. We should GC them as long as the container name starts with k8s.
I saw it today on a brand new cluster. On Mar 12, 2015 12:58 PM, "Yu-Ju Hong" wrote:
that should be expected. The first Kubelet that comes up is the Kubelet we have built-in the Container VM image and it doesn't do the new names yet.
No, it keeps happening, even later. I manually updated kubelet, it keeps popping up On Mar 12, 2015 1:10 PM, "Victor Marmol" wrote:
taking a look
indicates what happened above: 1) When node first bootstrap, it has containers running with old format 2) kubelet is upgrade, and kill old containers 3) GC periodically queries the dead containers with k8s prefix, and found those containers with old format, and throw log message. How to fix? just GC them.
Or simply lower down the log level to VLOG(4) since it should be a short-term thing.
Just sent out with the fix. The issue was container GC. Old Kubelet created containers New Kubelet comes up and kills old containers GC runs, but since the containers don't have a UID we don't remove them (repeat) Verified on a new cluster.


Question: intr := interrupt.New(nil, cancel) var result *corev1.Pod err := intr.Run(func() error { <del> ev, err := watchtools.UntilWithSync(ctx, lw, &corev1.Pod{}, preconditionFunc, func(ev watch.Event) (bool, error) { </del> <ins> ev, err := watchtools.UntilWithSync(ctx, lw, &corev1.Pod{}, nil, func(ev watch.Event) (bool, error) { </ins> switch ev.Type { case watch.Deleted: return false, errors.NewNotFound(schema.GroupResource{Resource: "pods"}, "")

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: return tabbedString(func(out io.Writer) error { w := NewPrefixWriter(out) w.Write(LEVEL_0, "Name:t%vn", ing.Name) <ins> printLabelsMultiline(w, "Labels", ing.Labels) </ins> w.Write(LEVEL_0, "Namespace:t%vn", ing.Namespace) w.Write(LEVEL_0, "Address:t%vn", loadBalancerStatusStringer(ing.Status.LoadBalancer, true)) def := ing.Spec.DefaultBackend

When command 'kubectl describe ing' Name: test-ing Namespace: practice Address: 10.10.01.10 Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found) TLS: test terminates Rules: Host Path Backends ---- ---- -------- / test-svc:80 (<none) Annotations: nginx Events: <noneResult has no information about labels. But Pod, Deployment, Service resources has label information like below. Name: test-room-test-mjsbz Namespace: test Priority: 0 Node: aks-test-test-test/22.22.222. Start Time: Fri, 09 Jul 2021 05:45:26 +0000 Labels: app=test pod-template-hash=test role=mediatest Why ingress has no information about labels? Actually when delete ingresss with labels is working correctly. Kubernetes version (use ): 1.18.14 Cloud provider or hardware configuration: AKS (Azure) OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig
/sig
/sig cli
/assign
/assign
/triage accepted /remove-kind bug /kind feature


Question: redirectWant: "http://example.com/redirected/target/", forwardedURI: "/proxy/node/node1:10250/redirect", }, <ins> "redirect abs use reqHost no host no scheme": { sourceURL:    "http://mynode.com/redirect", transport:    emptyHostAndSchemeTransport, redirect:     "http://10.0.0.1:8001/redirected/target/", redirectWant: "http://10.0.0.1:8001/proxy/node/node1:10250/redirected/target/", forwardedURI: "/proxy/node/node1:10250/redirect", reqHost:      "10.0.0.1:8001", }, </ins> "source contains the redirect already": { input:        `<pre><a href="kubelet.log">kubelet.log</a><a href="http://foo.com/proxy/node/node1:10250/google.log">google.log</a></pre>`, sourceURL:    "http://foo.com/logs/log.log",

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When accessing a service (e.g., ) by using the Apiserver proxy feature and the service returns with header, such as The returned header does not preserve the path if the service sent a redirect with an absolute path. Depending on the cluster, the header written by the service itself might have been and the apiserver proxy did rewrite the host:port to match the correct values. Unfortunately, it did not rewrite the path segment. What you expected to happen: I would expect the apiserver to rewrite the path segment of the response header so that it reflects the fact that it's proxied under the path: How to reproduce it (as minimally and precisely as possible): The path was not preserved. Anything else we need to know?: To work around this issue, a service may return strictly a relative url in the header such as: The feature to rewrite headers is quite common. For instance, Apache Httpd does it thanks to: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: custom_ OS (e.g. from /etc/os-release): Kernel (e.g. ):
/sig api-machinery /sig network
/assign
Thanks for the bug report!
GitHub didn't allow me to assign the following users: roycaihw. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err == nil { t.Fatalf("Random doesn't support fieldValidation, yet no error found") } <ins> err = fieldValidationVerifier.HasSupport(schema.GroupVersionKind{Group: "", Version: "v1", Kind: "List"}) if err == nil { t.Fatalf("List does not support fieldValidation, yet no error found") } </ins> } type EmptyOpenAPI struct{}

kubectl apply performs validation before the apply. Non-CRD resources (e.g. pods / deployments) don't need CRD schema for validation, but if we use resource, kubectl calls . In my Kubernetes cluster, administrator doesn't allow users to list CRDs, so user cannot apply non-CRD resources (e.g. pods / deployments) with resource. doesn't happen because we're going to apply two Deployments. should not happen because we're going to apply two Deployments. No response <details</details<details</details<details</details<details</details<details</details<details</details
/sig cli
/triage accepted /assign
/priority backlog


Question: continue } if lp.Protocol == "udp" { <del> isIPv6 := svcInfo.clusterIP.To4() != nil </del> <ins> isIPv6 := utilproxy.IsIPv6(svcInfo.clusterIP) </ins> utilproxy.ClearUDPConntrackForPort(proxier.exec, lp.Port, isIPv6) } replacementPortsMap[lp] = socket

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: There is a bug in ipv6, see: What you expected to happen: The judgement logic should be: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig network /area ipv6
/assign cc


Question: <ins> /* Copyright 2019 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package dynamiccertificates import ( "crypto/tls" "fmt" "io/ioutil" "sync/atomic" "time" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/util/workqueue" "k8s.io/klog" ) // DynamicFileServingContent provides a CertKeyContentProvider that can dynamically react to new file content type DynamicFileServingContent struct { name string // certFile is the name of the certificate file to read. certFile string // keyFile is the name of the key file to read. keyFile string // servingCert is a certKeyContent that contains the last read, non-zero length content of the key and cert servingCert atomic.Value listeners []Listener // queue only ever has one item, but it has nice error handling backoff/retry semantics queue workqueue.RateLimitingInterface } var _ Notifier = &DynamicFileServingContent{} var _ CertKeyContentProvider = &DynamicFileServingContent{} var _ ControllerRunner = &DynamicFileServingContent{} // NewDynamicServingContentFromFiles returns a dynamic CertKeyContentProvider based on a cert and key filename func NewDynamicServingContentFromFiles(purpose, certFile, keyFile string) (*DynamicFileServingContent, error) { if len(certFile) == 0 || len(keyFile) == 0 { return nil, fmt.Errorf("missing filename for serving cert") } name := fmt.Sprintf("%s::%s::%s", purpose, certFile, keyFile) ret := &DynamicFileServingContent{ name:     name, certFile: certFile, keyFile:  keyFile, queue:    workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), fmt.Sprintf("DynamicCABundle-%s", purpose)), } if err := ret.loadServingCert(); err != nil { return nil, err } return ret, nil } // AddListener adds a listener to be notified when the serving cert content changes. func (c *DynamicFileServingContent) AddListener(listener Listener) { c.listeners = append(c.listeners, listener) } // loadServingCert determines the next set of content for the file. func (c *DynamicFileServingContent) loadServingCert() error { cert, err := ioutil.ReadFile(c.certFile) if err != nil { return err } key, err := ioutil.ReadFile(c.keyFile) if err != nil { return err } if len(cert) == 0 || len(key) == 0 { return fmt.Errorf("missing content for serving cert %q", c.Name()) } // Ensure that the key matches the cert and both are valid _, err = tls.X509KeyPair(cert, key) if err != nil { return err } newCertKey := &certKeyContent{ cert: cert, key:  key, } // check to see if we have a change. If the values are the same, do nothing. existing, ok := c.servingCert.Load().(*certKeyContent) if ok && existing != nil && existing.Equal(newCertKey) { return nil } c.servingCert.Store(newCertKey) for _, listener := range c.listeners { listener.Enqueue() } return nil } // RunOnce runs a single sync loop func (c *DynamicFileServingContent) RunOnce() error { return c.loadServingCert() } // Run starts the controller and blocks until stopCh is closed. func (c *DynamicFileServingContent) Run(workers int, stopCh <-chan struct{}) { defer utilruntime.HandleCrash() defer c.queue.ShutDown() klog.Infof("Starting %s", c.name) defer klog.Infof("Shutting down %s", c.name) // doesn't matter what workers say, only start one. go wait.Until(c.runWorker, time.Second, stopCh) // start timer that rechecks every minute, just in case.  this also serves to prime the controller quickly. _ = wait.PollImmediateUntil(FileRefreshDuration, func() (bool, error) { c.queue.Add(workItemKey) return false, nil }, stopCh) // TODO this can be wired to an fsnotifier as well. <-stopCh } func (c *DynamicFileServingContent) runWorker() { for c.processNextWorkItem() { } } func (c *DynamicFileServingContent) processNextWorkItem() bool { dsKey, quit := c.queue.Get() if quit { return false } defer c.queue.Done(dsKey) err := c.loadServingCert() if err == nil { c.queue.Forget(dsKey) return true } utilruntime.HandleError(fmt.Errorf("%v failed with : %v", dsKey, err)) c.queue.AddRateLimited(dsKey) return true } // Name is just an identifier func (c *DynamicFileServingContent) Name() string { return c.name } // CurrentCertKeyContent provides serving cert byte content func (c *DynamicFileServingContent) CurrentCertKeyContent() ([]byte, []byte) { certKeyContent := c.servingCert.Load().(*certKeyContent) return certKeyContent.cert, certKeyContent.key } </ins>

When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ResourceParameters: &ResourceConfig{}, } if err := m.cgroupManager.Destroy(containerConfig); err != nil { <ins> klog.Warningf("failed to delete cgroup paths for %v : %v", podCgroup, err) </ins> return fmt.Errorf("failed to delete cgroup paths for %v : %v", podCgroup, err) } return nil

What happened: Housekeeping sync loop finds an orphaned pod and tries to clean it up. The kubepod slice, as seen below, is not removed by with no indication as to why. This leads to the docker scope timing out and entering a failed state. The returned error(s) from runc are swallowed making debugging difficult. What you expected to happen: As per other areas of the sync loop, useful errors are logged at debug level due to the async calls. How to reproduce it (as minimally and precisely as possible): N/A Anything else we need to know?: N/A Environment: Kubernetes version (use ): master
/sig node
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Have you seen the comment above pcm.Destroy() call ? Did you have a chance to look at kubelet log to see if the above was true ?
/remove-lifecycle stale
This would give us some clue about the error:
yes, definitely looked! In this specific situation, the pods are never retried. What we see happen with these pods: ADD -RECONCILE -Started Volumes -Created slice -No sandbox for pod -Started libcontainer -ContainerStarted -ContainerStarted -RECONSILE -ContainerDied -RECONCILE -Unmounts -Killing unwanted pod -Orphaned removing pod cgroups -container kill failed because of 'container not found' or 'no such process' -cleanup: failed to unmount secrets -ContainerDied -Container garbage collection succeeded -SyncLoop (SYNC): 141 pods (including this one) -stopping timed out. Killing.

-Stopped libcontainer -Removed slice -DELETE -REMOVE -err: pod not found Destroy all cgroups of the pod that should not be running, by first killing all the attached processes to these cgroups. We ignore errors thrown by the method, as the housekeeping loop would again try to delete these unwanted pod cgroups Despite running, the slice is not removed and we do not log or indicate why. Subsequently, the pod is NOT determined to be either an active pod or for some reason not in the cgroup list that is passed into kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) during housekeeping loops.

A more detailed look, which I believe is accurate ;): housekeepingCh -HandlePodCleanups -cgroupPods=UID+CGroupName=GetAllPodsFromCgroups -allPods=GetPodsAndMirrorPods -activePods=filterOutTerminatedPods -cleanupOrphanedPodCgroups I do not think desiredPods and runningPods are relevant here.. So either: a) the pod is not found and to allPods (seems unlikely) but we can't know for sure.. b) the pod is incorrectly determined to be active via activePods which again we can not determine.. Any pod that is not Failed, Succeeded or has a DeletionTimestamp set and a status of Terminated or Waiting - anything else returns false and is deemed active c) the pod cgroup is not found in m.subsystems.MountPoints and to cgroupPods - unlikely, but could be a bug but the root of this issue is the fact that all of the scopes and slices exist and keep building up until things start failing... GetAllPodsFromCgroups could use some logging... Maybe a V 5 log that prints out foundPods at the end... From there, they're passed into the cleanup: kl.cleanupOrphanedPodCgroups(cgroupPods, activePods) 1) All activePod UIDs are put into a podSet 2) A range loop of uid and cgroup name from cgroupPods looks to see if the uid from cgroupPods also exists in the podSet (active pods) 3) If the uid from cgroupPods exists in the podSet we skip out of the loop 4) If it doesn't, the pod is orphaned (not active w/ cgroups on the filesystem) and we call pcm.Destroy(val) on the cgroup name. Doesn't make sense why pcm.Destroy(val) ignores errors... it would be helpful!

tryKillingCgroupProcesses logs any error and also returns the an error which is always swallowed.. we don't see any issues here More problematic, the call to cgroupManager.Destroy only returns an error which would be swallowed... Could probably use a similar level 3 log here like what you have for tryKillingCgroupProcesses because as you can see, were totally blind here...
your change is of a similar nature to what I proposed in my PR for this issue, just a level higher. I think there are a few areas noted above the could help further debug this.. Still haven't been able to get my PR merged with additional logging.
The following code was relatively new (compared to when you logged this issue):
Indeed... I looked into it a while ago and those changes are new via: - Could definitely make a different if the issue is related to pod getting filtered out from activePods when they have not been fully cleaned up yet on the subsequent housekeeping syncloop runs.
I think is promising, but I still think it makes sense to add some logging here. Do you want want to proceed with my open PR, your suggestion in or both?
You can proceed with your PR.


Question: mock := newMockLister(c.successes, c.failures, v1alpha1.InitializerConfigurationList{}, t) manager := NewInitializerConfigurationManager(mock) manager.interval = 1 * time.Millisecond <del> manager.Run(mock.stopCh) <-mock.stopCh </del> <ins> for i := 0; i < 1+c.successes+c.failures; i++ { manager.sync() } </ins> _, err := manager.Initializers() if err != nil && c.expectReady { t.Errorf("case %s, expect ready, got: %v", c.name, err)

Found in I can reproduce locally using
/assign (via git blame)
cc
Thanks. I will take a look.
Got a fix locally. Testing it.


Question: ) // serviceConfig is a deserialized form of the config file format which ConfigSourceFile accepts. <ins> // TODO: this is apparently untested; is it used? </ins> type serviceConfig struct { Services []struct { Name      string   `json: "name"` Port      int      `json: "port"` Endpoints []string `json: "endpoints"` <del> } `json: "service"` </del> <ins> } `json:"service"` </ins> } // ConfigSourceFile periodically reads service configurations in JSON from a file, and sends the services and endpoints defined in the file to the specified channels.

For example: struct field tag not compatible with reflect.StructTag.Get missing argument for Errorf("%v"): format reads arg 2, have only 1 args range variable pod enclosed by function struct field tag not compatible with reflect.StructTag.Get arg minion for printf verb %s of wrong type: unreachable code unreachable code
Those all look really important
I'm going to fix this unless I hear that someone else is working on this in the next couple minutes.


Question: type updateNamespaceFunc func(kubeClient clientset.Interface, namespace *api.Namespace) (*api.Namespace, error) // retryOnConflictError retries the specified fn if there was a conflict error <ins> // it will return an error if the UID for an object changes across retry operations. </ins> // TODO RetryOnConflict should be a generic concept in client code func retryOnConflictError(kubeClient clientset.Interface, namespace *api.Namespace, fn updateNamespaceFunc) (result *api.Namespace, err error) { latestNamespace := namespace

See for full dumps. The line number of corresponds to:
From a quick look, perhaps the namespace object returned from retryOnConflictError() just above no longer has a DeletionTimestamp and thus "*namespace.DeletionTimestamp" triggers the panic?
Thanks for the report, I will investigate.


Question: - /run.sh resources: limits: <del> cpu: 0.5 </del> <ins> cpu: 0.1 </ins> env: - name: MAX_HEAP_SIZE value: 512M

The namespace seems to be hard coded :
cc:
Thanks. We can improve it after v1 release.


Question: # See the License for the specific language governing permissions and # limitations under the License. <del> ARG BASEIMAGE </del> ARG RUNNERIMAGE <del> FROM ${BASEIMAGE} as debbase </del> FROM ${RUNNERIMAGE} <del> # This is a dependency for `kubectl diff` tests COPY --from=debbase /usr/bin/diff /usr/local/bin/ COPY --from=debbase /lib/x86_64-linux-gnu/libc.so.6 /lib/x86_64-linux-gnu COPY --from=debbase /lib/x86_64-linux-gnu/libpthread.so.0 /lib/x86_64-linux-gnu COPY --from=debbase /lib64/ld-linux-x86-64.so.2 /lib64 </del> COPY cluster /kubernetes/cluster COPY ginkgo /usr/local/bin/ COPY e2e.test /usr/local/bin/

N/A 07/18 02:38 PDT No response /sig release
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Slack thread:
/sig cli


Question: }}, slice.Endpoints) } <del> func TestEnsureSetupManagedByAnnotation(t *testing.T) { serviceName := "testing-1" testCases := map[string]struct { serviceAnnotation   string startingSliceLabels map[string]string expectedSliceLabels map[string]string }{ "already-labeled": { serviceAnnotation: "foo", startingSliceLabels: map[string]string{ discovery.LabelServiceName: serviceName, discovery.LabelManagedBy:   controllerName, }, expectedSliceLabels: map[string]string{ discovery.LabelServiceName: serviceName, discovery.LabelManagedBy:   controllerName, }, }, "already-annotated": { serviceAnnotation: managedBySetupCompleteValue, startingSliceLabels: map[string]string{ discovery.LabelServiceName: serviceName, discovery.LabelManagedBy:   "other-controller", }, expectedSliceLabels: map[string]string{ discovery.LabelServiceName: serviceName, discovery.LabelManagedBy:   "other-controller", }, }, "missing-and-extra-label": { serviceAnnotation: "foo", startingSliceLabels: map[string]string{ discovery.LabelServiceName: serviceName, "foo":                      "bar", }, expectedSliceLabels: map[string]string{ discovery.LabelServiceName: serviceName, discovery.LabelManagedBy:   controllerName, "foo":                      "bar", }, }, "different-service": { serviceAnnotation: "foo", startingSliceLabels: map[string]string{ discovery.LabelServiceName: "something-else", }, expectedSliceLabels: map[string]string{ discovery.LabelServiceName: "something-else", }, }, } for name, testCase := range testCases { t.Run(name, func(t *testing.T) { client, esController := newController([]string{"node-1"}) ns := metav1.NamespaceDefault service := createService(t, esController, ns, serviceName, testCase.serviceAnnotation) endpointSlice := &discovery.EndpointSlice{ ObjectMeta: metav1.ObjectMeta{ Name:      "testing", Namespace: ns, Labels:    testCase.startingSliceLabels, }, } err := esController.endpointSliceStore.Add(endpointSlice) if err != nil { t.Fatalf("Expected no error adding EndpointSlice: %v", err) } _, err = client.DiscoveryV1beta1().EndpointSlices(ns).Create(endpointSlice) if err != nil { t.Fatalf("Expected no error creating EndpointSlice: %v", err) } esController.ensureSetupManagedByAnnotation(service) updatedService, err := client.CoreV1().Services(ns).Get(service.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("Expected no error getting Service: %v", err) } if updatedService.Annotations[managedBySetupAnnotation] != managedBySetupCompleteValue { t.Errorf("Expected managedBySetupAnnotation: %+v, got: %+v", managedBySetupCompleteValue, updatedService.Annotations[managedBySetupAnnotation]) } updatedSlice, err := client.DiscoveryV1beta1().EndpointSlices(ns).Get(endpointSlice.Name, metav1.GetOptions{}) if err != nil { t.Fatalf("Expected no error getting EndpointSlice: %v", err) } if !reflect.DeepEqual(updatedSlice.Labels, testCase.expectedSliceLabels) { t.Errorf("Expected labels: %+v, got: %+v", updatedSlice.Labels, testCase.expectedSliceLabels) } }) } } </del> // Test helpers func standardSyncService(t *testing.T, esController *endpointSliceController, namespace, serviceName, managedBySetup string) {

What happened: kube-controller-manager panic for What you expected to happen: no panic. How to reproduce it (as minimally and precisely as possible): No idea. I guess it's hard to reproduce. Anything else we need to know?: Found from here: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign cc Should we do something for v1.17? (v1.17 code freeze now, but still have the opportunity to fix) can be removed safely in v1.18, right?
/sig scheduling
Working on this in /sig network /priority critical-urgent /assign
Sorry, I didn't realize it. I should have a search before firing an issue. I recommend adding this issue to your PR. e.g. `
Updated, thanks!
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This seems to be writing to a cached (should be const) service. I think we need to DeepCopy() it before we mutate the Go type. does that sound right? Is this still our best state of the art?
yes, copy before change
we have a tool to catch this (the cache mutation detector), but it got disabled in our test runs two years ago (xref )
yes great catch - I was always convinced this is enabled...


Question: MountPath: "/subpath_mount", }, { <del> Name:      "workdir2", </del> <ins> Name:      "workdir1", </ins> MountPath: "/volume_mount", }, },

/kind cleanup /sig storage /sig node /sig testing Description: Change e2e tests to use EmptyDir instead of HostPath as pre-requisite to promoting VolumeSubpathEnvExpansion to Conformance. Related Issues: Conformance Tests:
/cc
/assign


Question: return isNodeReady(&nodeList.Items[0]) } <del> func systemctlDaemonReload() error { cmd := "systemctl daemon-reload" _, err := runCommand("sh", "-c", cmd) return err } var ( dbusConfPath = "/etc/systemd/system/dbus.service.d/k8s-graceful-node-shutdown-e2e.conf" dbusConf     = ` [Unit] RefuseManualStart=no RefuseManualStop=no [Service] KillMode=control-group ExecStop= ` ) func overlayDbusConfig() error { err := os.MkdirAll(filepath.Dir(dbusConfPath), 0755) if err != nil { return err } err = os.WriteFile(dbusConfPath, []byte(dbusConf), 0644) if err != nil { return err } return systemctlDaemonReload() } func restoreDbusConfig() error { err := os.Remove(dbusConfPath) if err != nil { return err } return systemctlDaemonReload() } </del> const ( // https://github.com/kubernetes/kubernetes/blob/1dd781ddcad454cc381806fbc6bd5eba8fa368d7/pkg/kubelet/nodeshutdown/nodeshutdown_manager_linux.go#L43-L44 podShutdownReason  = "Terminated"

All the crio serial jobs (presubmits and periodics) are not displaying the test grid results and timing out. periodics: node-kubelet-serial-crio presubmits: pull-kubernetes-node-kubelet-serial-crio-cgroupv1 pull-kubernetes-node-kubelet-serial-crio-cgroupv2 At least 6 months If you look into the artifacts the jobs are running but there seems to be hitting a timeout in the e2e_node tests. This causes the grid to not post the results so it marks the job as failed. We tried increasing the timeout to 6 hours from 4 hours but the timeout still occurs. Something is not right and we could use some help on figuring out why this can be happening. is one example that works without issue. /sig node testing
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
Looks like our recent PR didn't fix this.
Looking into the serial logs, it looks like these tests are hitting OOM. [ 3276.] Out of memory: Killed process (dd) total-vm:10490032kB, anon-rss:3218048kB, file-rss:80kB, shmem-rss:0kB, UID:0 pgtables:6352kB oomscoreadj:1000 [ 3276.] Out of memory: Killed process (dd) total-vm:10490032kB, anon-rss:3218048kB, file-rss:80kB, shmem-rss:0kB, UID:0 pgtables:6352kB oomscoreadj:1000
/reopen Memory limits did not help..
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue.

<detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Bumping machine size didn't fix this.
/cc :eyes:
dd is run in a couple of tests as far as I can see: running dd to simulate disk pressure, is similar. So, I'd start with The only Serial test I can find is this one: I've removed label from it to see it helps the test, here is a test PR: Let's see how it goes.
This looks quite dangerous and explains why bigger instances and limits didn't help: It tries to allocate 100Gb of memory.
Successful run has 'Out of memory: Killed process' as well, e.g.
It looks like OOMKiller message is expected and doesn't impact the issue. It's produced by the test case mentioned above. Without the test case the job is still getting timeouted and doesn't produce artifacts. Here is an example of failed and timeouted job run without the test case:
I don't think there is something wrong with long running tests. There are a lot of slow tests in the job and if infra is loaded with other tasks all of the tests would run slower. Here is a list of test durations for successful and failed job runs. I don't see anything that can't be explained by the busyness of the infra. Please correct me if I'm wrong here. successful run ()failed run (): One strange thing though is that the total time for the failed test(.577 sec / 60 = 365.2596m) is less than the job timeout (420m). Any ideas why the job was timed out?
Without slow tests finished in around 1h 25m and produced valid artifacts.
Without slow tests and with OOMKiller test the job still works ok:
With all tests the issue persisted despite the 10h timeout: I'll start bisecting. Hopefully will spot the test that causes everything to slow down.
One thing I noticed is that these tests are running two times. You can search for the start tests banner and see that we restart if the test failed.
One of the failures in the failed link was because kubelet restart took more time than ImageMaximumGCAge of 1 minute, is this expected ?

Attempt to fix this
Looking into what tests take the longest, I notice that the *Manager jobs are all pretty beefy. WDYT of moving these to a separate serial job?
An interesting datapoint I found between containerd serial and crio serial: in containerd: On crio, it takes 974 seconds.
Running this test on gcp with crio I don't see why it is taking up that long. I think that there is something else going on in this test. In the tests that were passing this test only take 14 seconds also.
With all but last 3 slow tests excluded the job still run more than 8h: Remaining slow tests: Resource-usage, Restart and SystemNodeCriticalPod In contrast the job with all slow tests excluded took only 1h25m: I'm not sure if the reason of slowness is in one of those 3 tests or it's something else. I'll remove them and run the job again.
I don't think they're running 2 times. The build log is duplicated if testrun fails, I guess. Look at the duration time for the same test case. They're exactly the same with a high precision, which wouldn't be possible for two different test runs.
Job run with only and slow tests enabled took more than 5 hours and didn't produce enough artifacts:
Successful run is executed on , failed is executed on . Could it be something related with this datapoint?
job is a current suspect. Test run with enabled and takes 1h 46 min and looks good: I'll try to enable more slow jobs with excluded job.
Job run with enabled resourceusage, systemnodecritical, log rotation, density and eviction tests takes 1h51m and looks good: So far so good. Side note: I'm not sure I understand why logrotation, density and eviction tests marked slow as they altogether take around 5 minutes.
Job run with all tests except of takes 2h and still looks good: I'll try to investigate test.
Job run with all tests except of and takes 2h 4m. and still looks good:
Can you point out to those runs, please? In my testing both pull jobs use n1-standard-4 and results don't differ.

I was referring to the job shared in the description
Job run with all tests except of takes 2h 9m and looks good: So, test case seems to slow down everything.
Nice! So we can just filter out that job then? Not sure if we have a slow lane for crio/containerd?
checking the code it seems it is performing an Ubuntu specific task. Is this applicable/needed also for Fedora?
is a test case in the test: We can either remove/comment out it or investigate why it slows down everything.
It looks like it restarts dbus and systemd, which theoretically should work on any linux with systemd and dbus. However, that comment could mean that it was never tested on anything except Ubuntu.
The Dbus test case was by Can you help us to investigate why test case slows everything down on Fedora/CoreOS ?
The test case was by and tested only with as crio pull jobs didn't exist at that time. However, it should be failing/slowing down ci jobs if they exist.
I opened up I talked with a few people at Red hat and it was suggested that this test is bad. Going to try just skipping it on fedora. Restarting dbus is not recommended
I asked as he has been working on gracefulshutdown and asked about a few of these problems on RHEL like OS. I think skipping should be fine for this.
Fixed remaining failed tests. Both jobs are green now:
/assign
Nice job


Question: import ( "fmt" "sort" <del> "time" </del> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime"

What happened: On a resource with managedFields populated, a no-op patch modified the serialization order of managedFields. What you expected to happen: No-op patches do not result in a write to etcd. How to reproduce it (as minimally and precisely as possible): contains: the initial ADDED event from kubectl MODIFIED events from kube-controller-manager one MODIFIED event corresponding to the kubectl patch (from resourceVersion 1704 in 1.txt to resourceVersion 1718 in 2.txt) shows the kubectl and kube-controller-manager managed field sets swapped order: second patch was a no-op ( is empty) Anything else we need to know?: This was easiest to reproduce with server-side apply enabled at 100% /assign
/sig api-machinery
/wg apply
Can you print the before/after fieldmanager? I can't quite figure out what the actual values are. The code that sorts these lists is located
updated to unified diff and attached:
If you explicitly set the field manager to something unique, does it happen every time?
hm, no, it's definitely an ordering swap
is not sorted properly, since should come before .
We seem to also sort it in a second place:
Theory: the first time it goes through, the time is accurate to the nanosecond; the second time, it's accurate to the second
Yeah OK I'm pretty sure that's it, both times are the same second. Serialization drops the fractional part.


Question: } } <del> func (c *cacheWatcher) nextBookmarkTime(now time.Time) (time.Time, bool) { // For now we return 2s before deadline (and maybe +infinity is now already passed this time) // but it gives us extensibility for the future(false when deadline is not set). </del> <ins> // bookmarkHeartbeatFrequency defines how frequently bookmarks should be // under regular conditions. const bookmarkHeartbeatFrequency = time.Minute func (c *cacheWatcher) nextBookmarkTime(now time.Time) time.Time { // We try to send bookmarks: // (a) roughly every minute // (b) right before the watcher timeout - for now we simply set it 2s before //     the deadline // The former gives us periodicity if the watch breaks due to unexpected // conditions, the later ensures that on timeout the watcher is as close to // now as possible - this covers 99% of cases. heartbeatTime := now.Add(bookmarkHeartbeatFrequency) </ins> if c.deadline.IsZero() { <del> return c.deadline, false </del> <ins> return heartbeatTime } if pretimeoutTime := c.deadline.Add(-2 * time.Second); pretimeoutTime.Before(heartbeatTime) { return pretimeoutTime </ins> } <del> return c.deadline.Add(-2 * time.Second), true </del> <ins> return heartbeatTime </ins> } func getEventObject(object runtime.Object) runtime.Object {

Currently, (best-effort) watch bookmarks are sent just right before the given watch is supposed to timeout: In we're adjusting how the size of watch cache is determined and make it dynamic and more duration-oriented (rather than based on some magic constants) - we're targeting to always keep 5 minutes of history in watch cache. With that change, we should get back to our bookmark policy: change it to (additionally to right before timeout) also send bookmarks every N seconds (setting N to slightly less than expected watch cache history period, so slightly less than X minutes we will use there) scale-test this change (to see if it doesn't introduce non-negligible overhead) if so permanently introduce this change /sig scalability api-machinery
/cc
/cc
/cc
I'm reverting as it broke bookmarks. I'm going to fix and re-submit for review hopefully later this week.


Question: return } <ins> // Redirect requests of the form "/{resource}/{name}" to "/{resource}/{name}/" // This is essentially a hack for https://github.com/GoogleCloudPlatform/kubernetes/issues/4958. // Note: Keep this code after tryUpgrade to not break that flow. if len(parts) == 2 && !strings.HasSuffix(req.URL.Path, "/") { w.Header().Set("Location", req.URL.Path+"/") w.WriteHeader(http.StatusMovedPermanently) return } </ins> proxy := httputil.NewSingleHostReverseProxy(&url.URL{Scheme: location.Scheme, Host: location.Host}) if transport == nil { prepend := path.Join(r.prefix, resource, id)

This works: but this does not: We should somehow emphasize the need for the trailing slash for now, and then work out what is going wrong (client side scripting?) to fix this (our fault in the proxy or Grafana's?)
Likewise for Kibana.
For the Kibana case, I observe a sequence of correctly proxied GETs e.g. to: but then I see: which fails because I assume what was really needed is: and a curl against that gives a pile of plausible Javascript.
After a little thought, I think what's going on here is that web clients can't tell that is a directory without the trailing slash. One thing we could potentially do to help is to redirect (301) from to , since I don't think the former is useful for anyone.
cc
How are we generating that URL? Can we append a "/" at the end while generating the URL? Doing a redirect or auto-adding "/" in the proxy for all requests could break other requests.
The URL is not generated anywhere. It is assumed that '/proxy/service/<service name' would work. , Nikhil Jindal wrote:
I discussed this with and To clarify, proxy server does not require a trailing slash. It works as expected (gives the same response with or without a trailing slash) The client code, on the other hand, expects it to be rendered on a path with a trailing slash and breaks when that is not the case. Ideally, we should fix the client code. For now, we decided to update proxy to send a 301 redirect from a path without a trailing slash to a path with a trailing slash. I will send a PR for this.
By "client code" do you mean things like Elasticsearch and Kibana?
Yes. The code being rendered on those paths.
I agree with what said, I just want to clarify that it's not necessarily a "broken" client, just one with different expectations. Adding the 301 redirect seems like an easy way to satisfy such clients and reduce user pain.
Not fixed because of
On GCE, the redirection is still not happening because the request is upgraded before we get a chance to redirect On localhost, the redirection works fine but we are unable to load the css and js files because of
Is proxy still supported in v1beta3? I vaguely remember that we wanted to remove it.

I tried this: but got NotFound error.
Access to cluster-level logging components depends on the proxy so avoid making ELBs. Elasticsearch and Kibana are accessed via the v1beta3 service proxy.
Although the trailing slashes are no longer required.


Question: return } <ins> // Redirect requests of the form "/{resource}/{name}" to "/{resource}/{name}/" // This is essentially a hack for https://github.com/GoogleCloudPlatform/kubernetes/issues/4958. // Note: Keep this code after tryUpgrade to not break that flow. if len(parts) == 2 && !strings.HasSuffix(req.URL.Path, "/") { w.Header().Set("Location", req.URL.Path+"/") w.WriteHeader(http.StatusMovedPermanently) return } </ins> proxy := httputil.NewSingleHostReverseProxy(&url.URL{Scheme: location.Scheme, Host: location.Host}) if transport == nil { prepend := path.Join(r.prefix, resource, id)

This works: but this does not: We should somehow emphasize the need for the trailing slash for now, and then work out what is going wrong (client side scripting?) to fix this (our fault in the proxy or Grafana's?)
Likewise for Kibana.
For the Kibana case, I observe a sequence of correctly proxied GETs e.g. to: but then I see: which fails because I assume what was really needed is: and a curl against that gives a pile of plausible Javascript.
After a little thought, I think what's going on here is that web clients can't tell that is a directory without the trailing slash. One thing we could potentially do to help is to redirect (301) from to , since I don't think the former is useful for anyone.
cc
How are we generating that URL? Can we append a "/" at the end while generating the URL? Doing a redirect or auto-adding "/" in the proxy for all requests could break other requests.
The URL is not generated anywhere. It is assumed that '/proxy/service/<service name' would work. , Nikhil Jindal wrote:
I discussed this with and To clarify, proxy server does not require a trailing slash. It works as expected (gives the same response with or without a trailing slash) The client code, on the other hand, expects it to be rendered on a path with a trailing slash and breaks when that is not the case. Ideally, we should fix the client code. For now, we decided to update proxy to send a 301 redirect from a path without a trailing slash to a path with a trailing slash. I will send a PR for this.
By "client code" do you mean things like Elasticsearch and Kibana?
Yes. The code being rendered on those paths.
I agree with what said, I just want to clarify that it's not necessarily a "broken" client, just one with different expectations. Adding the 301 redirect seems like an easy way to satisfy such clients and reduce user pain.
Not fixed because of
On GCE, the redirection is still not happening because the request is upgraded before we get a chance to redirect On localhost, the redirection works fine but we are unable to load the css and js files because of
Is proxy still supported in v1beta3? I vaguely remember that we wanted to remove it.

I tried this: but got NotFound error.
Access to cluster-level logging components depends on the proxy so avoid making ELBs. Elasticsearch and Kibana are accessed via the v1beta3 service proxy.
Although the trailing slashes are no longer required.


Question: // TaintNodeUnreachable would be automatically added by node controller // when node becomes unreachable (corresponding to NodeReady status ConditionUnknown) // and removed when node becomes reachable (NodeReady status ConditionTrue). <del> TaintNodeUnreachable = "node.alpha.kubernetes.io/unreachable" </del> <ins> TaintNodeUnreachable = "node.kubernetes.io/unreachable" // DeprecatedTaintNodeUnreachable is the deprecated version of TaintNodeUnreachable. // It is deprecated since 1.9 DeprecatedTaintNodeUnreachable = "node.alpha.kubernetes.io/unreachable" </ins> // When feature-gate for TaintBasedEvictions=true flag is enabled, // TaintNodeOutOfDisk would be automatically added by node controller

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When discussing issue in , I noticed: although is using key , it has actually been changed to in for quite a while. I am thinking we may need to fix this inconsistency in 1.9, by moving out of alpha. Anything else we need to know?: Task tracker: [x] update [x] update kubernetes doc
/sig scheduling


Question: // It returns a list of containers, or an error if one occurs. func eventToPods(ev watch.Event) ([]api.BoundPod, error) { pods := []api.BoundPod{} <ins> if ev.Object == nil { return pods, nil } </ins> boundPods, ok := ev.Object.(*api.BoundPods) if !ok { return pods, errors.New("unable to parse response as BoundPods")

To reproduce: Start a cluster via hack/local-up- Start some pods Kill the cluster "docker ps -a" should show the pod containers running still Start the cluster again "etcdctl ls /registry" should show that there is no boundpods directory -- only minions and services "docker ps -a" should show the old pod containers still running Start a new pod After, say, 10s, "docker ps -a" should show the old containers dead /cc
I suspect the cause of this might be : which guarantees that the pod sync logic is never called if is nil. Need to get a repro to confirm.


Question: server.Close() } } <ins> func TestGetServerPreferredResourcesRetries(t *testing.T) { stable := unversioned.APIResourceList{ GroupVersion: "v1", APIResources: []unversioned.APIResource{ {Name: "pods", Namespaced: true, Kind: "Pod"}, }, } beta := unversioned.APIResourceList{ GroupVersion: "extensions/v1", APIResources: []unversioned.APIResource{ {Name: "deployments", Namespaced: true, Kind: "Deployment"}, }, } response := func(numErrors int) http.HandlerFunc { var i = 0 return func(w http.ResponseWriter, req *http.Request) { var list interface{} switch req.URL.Path { case "/apis/extensions/v1beta1": if i < numErrors { i++ w.WriteHeader(http.StatusInternalServerError) return } list = &beta case "/api/v1": list = &stable case "/api": list = &unversioned.APIVersions{ Versions: []string{ "v1", }, } case "/apis": list = &unversioned.APIGroupList{ Groups: []unversioned.APIGroup{ { Name: "extensions", Versions: []unversioned.GroupVersionForDiscovery{ {GroupVersion: "extensions/v1beta1"}, }, PreferredVersion: unversioned.GroupVersionForDiscovery{ GroupVersion: "extensions/v1beta1", Version:      "v1beta1", }, }, }, } default: t.Logf("unexpected request: %s", req.URL.Path) w.WriteHeader(http.StatusNotFound) return } output, err := json.Marshal(list) if err != nil { t.Errorf("unexpected encoding error: %v", err) return } w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusOK) w.Write(output) } } tests := []struct { responseErrors  int expectResources int expectedError   func(err error) bool }{ { responseErrors:  1, expectResources: 2, expectedError: func(err error) bool { return err == nil }, }, { responseErrors:  2, expectResources: 1, expectedError:   IsGroupDiscoveryFailedError, }, } for i, tc := range tests { server := httptest.NewServer(http.HandlerFunc(response(tc.responseErrors))) defer server.Close() client := NewDiscoveryClientForConfigOrDie(&restclient.Config{Host: server.URL}) got, err := client.ServerPreferredResources() if !tc.expectedError(err) { t.Errorf("case %d: unexpected error: %v", i, err) } if len(got) != tc.expectResources { t.Errorf("case %d: expect %d resources, got %#v", i, tc.expectResources, got) } server.Close() } } </ins>

Split off from , the thing that actually caused the flakes throughout the system (as opposed to just failing the TPR test) is (https://k8s-) Which I guess means that the discovery client got a list, but then the TPR was gone when it tried to get details? This is not an emergency since we've marked the test flaky, but we should fix it.
The print out of the error message included the message, which was generic 404. So I believe returned a 404 without a JSON body.
Yes. The discovery client returns a partial result when this error occurs, see . I think this behavior is correct. Caller should look into the error and ignore it or retry the discovery.


Question: ] }` <ins> // This token is known to apiserver and its group is `system:masters`. // See also the function `generateTokenFile` in `test/e2e_node/services/apiserver.go`. kubeConfig := fmt.Sprintf(` apiVersion: v1 kind: Config users: - name: node-problem-detector user: token: %s clusters: - cluster: server: %s insecure-skip-tls-verify: true name: local contexts: - context: cluster: local user: node-problem-detector name: local-context current-context: local-context `, framework.TestContext.BearerToken, framework.TestContext.Host) </ins> ginkgo.By("Generate event list options") selector := fields.Set{ "involvedObject.kind":      "Node",

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Last successful run (23 Oct 2020, 10:07 PDT): First unsuccessful run (23 Oct 2020, 12:08 PDT): Testgrid link: Reason for failure: Anything else we need to know: presubmit job in repository also started to fail in the same fashion. Attempted to debug it myself, found out the following stuff: last successful run was accessing the apiserver via HTTP: all unsuccessful runs attempted to access the apiserver via HTTPS but to no avail: perused the history of to find out that the PR is almost certainly the reason of the failures (or rather uncovered some problem with the NPD or its tests): first, it coincides with the start of the failures of , second, we changed the default hostname of the apiserver in node tests in that PR. Judging by the code and the changes in the aforementioned PR I think that a quick mitigation would be to add a flag to the of these jobs. /assign /sig node /cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Opened for a mitigation that would make those jobs stop failing.
This won't work, unfortunately. Copying response:
I have filed to address this issue, but it seems I could not run the test in advance to verify my patch, did you know how to manually trigger the presubmit job?
Unfortunately I don't :/. My PR was also a wild guess. could you take a look at the proposed fix? Thanks once again for your help - really appreciated :).
the submission of hasn't fixed the problems, unfortunately :/. Sample failing run: We still have some peculiar problem related to health checking: and now NPD panics, too:
Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. /reopen
Reopened this issue.

<detailsIn response to : Thanks for your report, I believe the reason is admission controller has been disabled in node e2e tests: We need to figure out a new approach now. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
filed . I hope we could completely address this issue this time.
/milestone v1.20


Question: continue } <ins> var azNames []string for key := range subnetsByAZ { azNames = append(azNames, key) } sort.Strings(azNames) </ins> var subnetIDs []string <del> for _, subnet := range subnetsByAZ { subnetIDs = append(subnetIDs, aws.StringValue(subnet.SubnetId)) </del> <ins> for _, key := range azNames { subnetIDs = append(subnetIDs, aws.StringValue(subnetsByAZ[key].SubnetId)) </ins> } return subnetIDs, nil

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws Anything else we need to know?: AWS NLBs support static IP addresses by associating an EIP per AZ. An already allocated EIP can be supplied with the . A user can then supply EIP allocation IDs in the service definition of kubernetes. By that the service will be available via those static IPs.
eip's for white listing are really important in the enterprise, we need this feature now, as a result we have to manually create nlb's, would really like to see this
This is also important for hosting providers, who might have huge numbers of apex domain DNS records pointed at EIPs. Associating an already-allocated EIP with a new Kubernetes-generated NLB would be huge!
This feature would save us weeks of work reimplementing the cloud controller just so we can preserve a special ELB that has static IP addresses... I really wish it was available
I'd like to contribute time to working on this. Is anyone else actively working on adding support?
That's awesome news. Thanks for working on this. Do you need assistance in testing this in an AWS environment? I might help out with that ...
Sure, the PR has been approved for testing so the binaries are available. I'm moving the code into our soak environment to get it some more exposure this week.
I've been trying with: But I only go a NLB without the EIPs deployed. Didn't find any errors in the logs. Do I have to enable a feature flag for it or kind of?
The reviewer found a typo where I may not be collecting the correct annotation. I'm waiting for the new build to complete and then rolling to our soak test. I'm passing through any AWS error and also error if the number of EIP and Subnets don't match. Let me get it online with the changes from code review and I'll dig in deeper. Also, are you using the binaries directly from the build system or just Kubernetes 1.12?
I checked with the fixes you provided yesterday. Works :) Created a service which spawns over 3AZs and supplied 3 EIP allocations. NLBs gets created correctly.
Can you also comment on the PR? I'm sure having multiple successful tests would be helpful to the reviewer.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hey, thanks for doing this, I was just looking for a way to associate one eip to a nlb, one question though why do you need one eip for each AZ? now the nlb creates a 1 public IP that balance to 3 AZ using traefik in my case when I deploy the service, thanks
Due to the fact that EIPs are represented by network hardware in AWS's datacenter you'd like to have redundant instances. If you run with only one EIP and the AZ of this EIP goes down, you won't be able to reach the (still up and running) traefik deployment in the other two AZs.
Hi, I've seen the PR () has been waiting for an update for three months. do you need a hand? do you want me to pick it up? I am really interested on this to be merged :smile:
Can you help write the tests, ? I'm completely lost there to be honest.
Hi I've found out that test-coverage on the aws provider is low and there is no tests on the so I can't find tests on NLB. I am going to open a new issue and work on improving coverage for that and hopefully I'll find out how to add tests for the new annotation. As you can see below current coverage for and is low and can't find tests for NLB:
Thanks for the help. I thought I was just missing being able to locate them.
+1 need this ASAP We are hosted on EKS so only official version supported. Any idea when this will be merged?
.... As a workaround, I'm just manually creating a second NLB (with EIPs) and changing the listeners to forward the traffic to the automatically created target groups (from the original svc-linked NLB).
Waiting for this as well on EKS. Thanks!
would be great to have that in place. thanks! I can confirm that the manual workaround by works fine, but it is hard to automate that flow.
+1 Waiting for this as well on EKS.

+1
I have created a node project that runs as a cronjob and monitors a SOURCE target group and clones the targets to a DESTINATION target group. So when you create a LoadBalancer of type 'nlb' in kubernetes, all you need to do is manually create a new 'nlb' in AWS Console, assign your EIP to your NLB for a specific 'subnet'(availability zone), then create a target group with no targets; use the ARN from this target group as the destination and use the ARN of the kubernetes target group as the SOURCE. More information here: Public docker image:
I have hard time getting this working with NLB using ingress controller : Even subnets are not taking effect here , its not passing my configurations in the API that creates the NLB kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: ingress-nginx ingress-nginx annotations: nlb "eipalloc-,eipalloc-,eipalloc-" "subnet-,subnet-" spec: type: LoadBalancer selector: ingress-nginx ingress-nginx ports: name: http port: 80 targetPort: http name: https port: 443 targetPort: https =======================================================``
I am also getting the same error can anyone help me with this ??
what version of k8s you are using ? this is not supported before 1.15 as far as I know so my issue was that I tried to implement on EKS
Hi i am using the latest version of k8 1.15 .5
I am not using EKS with my solution but the problem is the same as you mentioned
New to EKS. Could you elaborate how the connectivity from NLB that we have created manually (as you mentioned ) to the ingress-nginx pod?
Has it worked for you in k8s 1.15 EKS?
How to get Allocation IDs?
my elastic eip ids cannot be found at service creation, so it just sits in pending forever. if I resort to listing the subnets on their own annotation, (not using subnet discovery) same thing. no matter what I do it seems to not find me elastic ip addresses. Are you sure nlb type is supported to work with Elastic IPs?

From the AWS docs it seem they are pretty heavily trying to discourage use of the nlb type (Cloud Provider Load Balancer) in favor of pushing the new AWS Load balancer.
See available annotations: https://kubernetes- If you have done that: what do your aws-lb-controller logs tell you? I guess there’s an access problem with your iam role. Is your aws-lb-controller iam autorized to list and modify eip’s? Are the eip annotations exactly correct, can they be found by the controller? The controller’s log stream should have some kind of an error there. I have successfully assigned eips using service annotations to provision an aws nlb, but pay attention, you need to install the aws-lb-controller yourself. The kubernetes built-in aws library does not support eips.
Here is the annotation needed for assigning a static Elastic IP to your NLB using the AWS Load Balancer (Ingress) Controller https://kubernetes- New Controller link https://kubernetes-


Question: <ins> /* Copyright 2023 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package main import ( "fmt" "os" "regexp" "strings" flag "github.com/spf13/pflag" kruntime "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/sets" "k8s.io/klog/v2" ) var ( typeSrc = flag.StringP("type-src", "s", "", "From where we are going to read the types") re      = regexp.MustCompile("`(bw+b)`") ) // kubeTypesMap is a map from field name to its tag name and doc. type kubeTypesMap map[string]kruntime.Pair func main() { flag.Parse() if *typeSrc == "" { klog.Fatalf("Please define -s flag as it is the api type file") } docsForTypes := kruntime.ParseDocumentationFrom(*typeSrc) rc := false for _, ks := range docsForTypes { typesMap := make(kubeTypesMap) for _, p := range ks[1:] { // skip the field with no tag name if p.Name != "" { typesMap[strings.ToLower(p.Name)] = p } } structName := ks[0].Name rc = checkFieldNameAndDoc(structName, "", ks[0].Doc, typesMap) for _, p := range ks[1:] { rc = checkFieldNameAndDoc(structName, p.Name, p.Doc, typesMap) } } if rc { os.Exit(1) } } func checkFieldNameAndDoc(structName, fieldName, doc string, typesMap kubeTypesMap) bool { rc := false visited := sets.Set[string]{} // The rule is: // 1. Get all back-tick quoted names in the doc // 2. Skip the name which is already found mismatched. // 3. Skip the name whose lowercase is different from the lowercase of tag names, //    because some docs use back-tick to quote field value or nil // 4. Check if the name is different from its tag name // TODO: a manual pass adding back-ticks to the doc strings, then update the linter to // check the existence of back-ticks nameGroups := re.FindAllStringSubmatch(doc, -1) for _, nameGroup := range nameGroups { name := nameGroup[1] if visited.Has(name) { continue } if p, ok := typesMap[strings.ToLower(name)]; ok && p.Name != name { rc = true visited.Insert(name) fmt.Fprintf(os.Stderr, "Error: doc for %s", structName) if fieldName != "" { fmt.Fprintf(os.Stderr, ".%s", fieldName) } fmt.Fprintf(os.Stderr, " contains: %s, which should be: %sn", name, p.Name) } } return rc } </ins>

Currently we have inconsistency in most of the API where the field name should be used in description so the generated docs match the API fields people actually set. A linter could be for checking To avoid confusing users
/good-first-issue
/triage accepted
/assign
Hey can you provide some more information for this issue
are you still working on this?
Hey feel free to take it. I am unassigning myself
/triage needs-information /remove-good-first-issue
/sig api-machinery


Question: return nil } <del> func (ds *dockerService) determinePodIPBySandboxID(uid string, sandbox *dockertypes.ContainerJSON) string { </del> <ins> func (ds *dockerService) determinePodIPBySandboxID(uid string) string { </ins> return "" }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: PR tries to reduce CNI calls for sandboxed containers: case is for Windows Server 2016 and for Windows Server version greater than 1709 which use Hyper-V containers. Currently, the CNI on Windows fetches the IP from the containers within the pods regardless of the mode. This should be done only in the non-sandbox mode where the IP of the actual container will be different than the IP of the sandbox container. from the same pod will share the network details of the sandbox container. when this mode is supported. However, for Windows 1709 and 1803 we didnt have the Namespace support in Windows which we are bringing in RS5. So we need this second call to replicate the DNS registry key to the Workload container. This is a platform requirement and removing this will break DNS for the workload containers. IP/Gateway/MAC is set separately than DNS. (Refer ) In Rs5 once we move the namespace model, we will update CRI and CNI to add endpoints and add it to the namespace. So the PR breaks dns for windows containers. We should revert it. /sig windows /kind bug
Since is headed for 1.11, this issue should be similarly milestone 1.11 labeled and approved. /milestone v1.11 /status approved-for-milestone
You must be a member of the github team to set the milestone. <detailsIn response to : Since is headed for 1.11, this issue should be similarly milestone 1.11 labeled and approved. /status approved-for-milestone Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
You must be a member of the github team to add status labels.
/milestone v1.11 /status approved-for-milestone
if you can drop an update here on status of the PR revert of PR , and if there's anything additional that might be happening like a re-fix of what was being attempted in the earlier PR, the 1.11 release team would appreciate the additional context.

PR has been reviewed by but still waiting for sig-node's approval
/priority critical-urgent /status in-progress
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process Note: This issue is marked as , and must be updated every 1 day during code freeze. Example update: <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: } pod, containerSpec = restoredPod, restoredContainer } // From this point , pod and container must be non-nil. gracePeriod := int64(minimumGracePeriodInSeconds) switch {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I tried to use virtlet with Kubernetes release 1.6.6, but a pod with a VM in it gets stuck in "ContainerCreating" state. This problem does not happen with Kubernetes release 1.6.3. The log of the responsible kubelet shows a panic when trying to restart the container. I do not know why the container is being restarted, but that should not cause a panic in any case. Here is an excerpt from the kubelet's log: It is easy to see a code path that can cause this panic. The offending line in is this: and there is indeed a way for to be nil at this point. Earlier in that function we see the line and there is no intervening statement that checks whether is ; ends with . What you expected to happen: I expected the virtlet to run, like it does with Kubernetes release 1.6.3. How to reproduce it (as minimally and precisely as possible): That would take some work, I hope the diagnosis above makes this moot. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration**: "bare metal" (really VMs, but who cares?) OS (e.g. from /etc/os-release): Ubuntu 16.04.2 LTS Kernel (e.g. ): Linux n11rdu1ctrl 4.4.0-34-generic -Ubuntu SMP Wed Jul 27 16:06:39 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: I installed Kubernetes and virtlet using some local Ansible Others: I am using a local fork of virtlet
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig node
Reiterating the mentions to trigger a notification: <detailsIn response to : </details
, did you do any update action against that Pods?

I upped the kubelet's to 4, and got an interesting clue when I tried the experiment again. Here is an excerpt from the more verbose log. Note the empty container name in the "Killing unwanted container" entry. That telling log entry comes from these lines in which is soon followed by .
I only created the pod, no updates.
Is there a running container with id something like when the panic occurs? A quick fix is to add checking whether containerSpec is nil as you mentioned. But it's weird that gets a container without name.
Did virtlet set correctly? Seems it is "", which caused the problem.
Create a PR () that checking containerSpec to avoid panic. And it seems there's no UT case for , I'll create related PRs for
The actual Docker containers are all gone now. But here is what I got from shortly before opening this issue: I will look into the question asked.


Question: return append(shell, editor), true } <del> func (e Editor) args(path string) []string { args := make([]string, len(e.Args)) copy(args, e.Args) if e.Shell { last := args[len(args)-1] args[len(args)-1] = fmt.Sprintf("%s %q", last, path) } else { args = append(args, path) } return args } // Launch opens the described or returns an error. The TTY will be protected, and // SIGQUIT, SIGTERM, and SIGINT will all be trapped. func (e Editor) Launch(path string) error { if len(e.Args) == 0 { return fmt.Errorf("no editor defined, can't open %s", path) } abs, err := filepath.Abs(path) if err != nil { return err } args := e.args(abs) cmd := exec.Command(args[0], args[1:]...) cmd.Stdout = os.Stdout cmd.Stderr = os.Stderr cmd.Stdin = os.Stdin klog.V(5).Infof("Opening file with editor %v", args) if err := (term.TTY{In: os.Stdin, TryDev: true}).Safe(cmd.Run); err != nil { if err, ok := err.(*exec.Error); ok { if err.Err == exec.ErrNotFound { return fmt.Errorf("unable to launch the editor %q", strings.Join(e.Args, " ")) } } return fmt.Errorf("there was a problem with the editor %q", strings.Join(e.Args, " ")) } return nil } </del> // LaunchTempFile reads the provided stream into a temporary file in the given directory // and file prefix, and then invokes Launch with the path of that file. It will return // the contents of the file after launch, any errors that occur, and the path of the

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--I was trying to use a different editor than notepad on windows to be used by by setting the environment variable. Doing this was non trivial and the following did not work all resulting in different errors. What happened: What you expected to happen: I should be able to use normal windows paths, either by quoting them, or not. the workaround is to use a short filename however short paths are optional and may not always be created. () How to reproduce it (as minimally and precisely as possible): try and set an editor on windows where there is a space in the name of the command. Anything else we need to know?: Environment: Kubernetes version (use ): v1.10.3 Cloud provider or hardware configuration: N/A this is client side issue OS (e.g. from /etc/os-release): windows 10 1803 Kernel (e.g. ): :-) Install tools: Others:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
In case anyone wants an answer, it needs two things: (or ) into system PATH environment variable EDITOR=notepad2 -w (or EDITOR=code -w)
Currently there is no way how to pass path with spaces on Windows with or environment variables. Pull Request addresses this issue. With the fix, you would be able to specify the following: and when is in your , you could do as well: or (this is current workaround for the problem)


Question: meta.Annotations[api.TolerationsAnnotationKey] = string(tolerationsAnnotation) } <del> func SetMasterNodeAffinity(meta *api.ObjectMeta) { </del> <ins> // SetNodeAffinity is a basic helper to set meta.Annotations[api.AffinityAnnotationKey] for one or more api.NodeSelectorRequirement(s) func SetNodeAffinity(meta *api.ObjectMeta, expr ...api.NodeSelectorRequirement) { </ins> nodeAffinity := &api.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &api.NodeSelector{ <del> NodeSelectorTerms: []api.NodeSelectorTerm{{ MatchExpressions: []api.NodeSelectorRequirement{{ Key: "kubeadm.alpha.kubernetes.io/role", Operator: api.NodeSelectorOpIn, Values: []string{"master"}, }}, }}, </del> <ins> NodeSelectorTerms: []api.NodeSelectorTerm{{MatchExpressions: expr}}, </ins> }, } affinityAnnotation, _ := json.Marshal(api.Affinity{NodeAffinity: nodeAffinity})

Right now we create 3 daemonsets for in , but will only give use image names for , so this feature is broken. We should only deploy one daemon set for the native architecture, but ensure it has the right label selectors, so we don't try to run pods on nodes of different architecture. We should also make sure all addons have architecture labels, and not just . If user is building heterogeneous clusters, the will need to take about deploying addons for additional architectures, we shouldn't pre-deploy multiple things knowing that they will most likely remain unschedulable.
I have the same issue as well. I tried to create a cluster with x86 (Intel NUC) and arm (Raspberry-Pi) and I see that the kube-proxy is crashing. Here's the log entry from kube-proxy Logs from kube-proxy exec user process caused "exec format error" Attached is the kube-proxy yaml file. I see that its picking up arm image Don't know how to fix this.