Question: } v2Mappings = append(v2Mappings, portMapping) <ins> } else { listener, err := buildListener(port, annotations, sslPorts) if err != nil { return nil, err } listeners = append(listeners, listener) </ins> } <del> listener, err := buildListener(port, annotations, sslPorts) if err != nil { return nil, err } listeners = append(listeners, listener) </del> } if apiService.Spec.LoadBalancerIP != "" {

What happened: What you expected to happen: kubernetes creating an AWS NLB for UDP How to reproduce it (as minimally and precisely as possible): Create service with annotation and port with Anything else we need to know?: AWS recently announced that UDP is available within their NLB Environment: Kubernetes version (use ): 1.11.9 Cloud provider or hardware configuration: AWS
/sig aws
/kind feature
/remove-kind bug
/milestone v1.16
You must be a member of the GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility. <detailsIn response to : /milestone v1.16 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.16
I am taking up this Feature Development. How can this be assigned to me?
/assign
+1
+1 I am experiencing the same issue. NLB also supports UDP_TCP type, any service using "nlb" should be able to mix TCP and UDP port configs.
Any progress? I can collaborate on this, if you need a hand.
+1
+1 On Thu, Jul 18, 2019 at 12:58 Adam Murphy <:
+1
I just ran into this. Any updates?
Attempting to fix it with
Thanks, The Changes looks good.
This should be closed I think.
This feature only for 1.16?
Could feature back ported into 1.15 1.14 1.13
Could someone review please?
/sig cloud-provider
Any update on this?
As we are in the code freeze and this issue is still not fixed, I'll move it to the 1.17 milestone. I'm also going to add the PR that fixes this issue to the milestone. If there are any concerns, please ping me or ping SIG-Release on Slack. /milestone v1.17
Do you have a timeline when this will be released?
This is a breaking issue in multiple versions but the is not prioritized. Does anyone have a workaround at the moment?
For AWS you can associate elastic IP with pod. Need make aws cli call.

If you need static private IP then you can create DaemonSet attach NIC with static IP as second NIC to EC2 instance and node some label (like ). Later you start pod with network, get node label, bound to required host IP. If you need static public and private IPs, then you can customize node-group init script and attach NIC with permanent private IP and Elastic IP at node startup. Later simple start pod with network
Bug triage for release 1.17 here. Since this issue has been tagged for the milestone, we would like to check it's status. Code freeze is on Nov. 14. Will this issue be resolved before then? If not, we should move it out of the milestone.
Hi bug triage team for 1.17, will this issue be resolved before code freeze on Nov. 14 (Next Thursday)?
/milestone v1.18 (v1.17 code freeze)
looking for a first issue to work, can I take this
gave some good feedback to the original PR. Does this need a reset with a fresh PR? I also have a need for this. I don't have a strong urge to write, or reuse, an operator for NLBs in AWS for UDP. This will eventually get done in k8s anyways, so I'd rather put the effort there.
BTW, you can always setup an NLB manually to forward UDP to a nodePort service on your k8s instances. The issue with that is that you have to pick a NodePort service for UDP before hand, and manually handle the NLB and the target groups. One cluster upgrade and your target group behind the NLB disappears since all of the instances get replaced. It's not ideal but in a pinch...
I don't mind updating a the PR, seems like a good first issue for me.
/assign
+1
+1
+1
+1
+1
Hello ! Bug Triage team here for the 1.18 release. This is a friendly reminder that code freeze is scheduled for 5 March, which is a few weeks from now. Is this issue still intended for milestone 1.18? Thanks in advance!

C/O Code freeze for release 1.18 is starting in 5 March (less than 1 week from now) and we want to ensure that each issue has a chance to be finished on time. Is this still on track for this deadline?
/remove-kind bug
Is there any work remaining to be done on this? Willing to contribute if so, would really love to see this make it into 1.18!
Thanks for the reply This has been a long running issue, and given that there has not been any recent activity, would it make more sense to once again move the milestone? We are 2 days away from code freeze (5 March).
I wonder what "activity" you are expecting.
We are now in code freeze for 1.18, so I am moving this to the next milestone. /milestone v1.19
Hello, as of now, I see is 65% completed. May we please have an update on estimated release date of UDP support for AWS NLB?
People have been trying to be constructive here. But the team doesn't even try to tell us what is pending. Sad.
Would also love to see this feature for EKS - and am willing to help out. Even if we could get it in 1.19 it would still take a long time to become available in EKS (which is currently at 1.15). So the sooner we can start work on this - the better :-)
+1 Also hope we could get in 1.19 and happy to help out!
any update on this pretty please ?
whats the update here?
From what I can see, the main issues that remain unresolved since the first PR last year: is still pending if you need a service that supports both TCP and UDP, e.g., DNS. DNS is also a more-complicated example because it supports TCP and UDP on the same port number, and that'll need more refactoring here, as AWS treats that as a single multi-protocol target. Edit: To be clear, not really a blocker for UDP support, but it's been raised in the discussion earlier. AWS NLB can only do TCP-based health checks (including HTTP and HTTPS), so your service needs to have a health-check TCP port listening. Current PRs silently assume it's on the same port number as the UDP service you're advertising.

If you're using , then it'll use the health check from  instead anyway. So this is probably only an issue for LoadBalancer Services with set. See "Things to know" on the .


Question: q.cond.Signal() } <ins> // Len returns the current queue length, for informational purposes only. You // shouldn't e.g. gate a call to Add() or Get() on Len() being a particular // value, that can't be synchronized properly. func (q *Type) Len() int { q.cond.L.Lock() defer q.cond.L.Unlock() return len(q.queue) } </ins> // Get blocks until it can return an item to be processed. If shutdown = true, // the caller should end their goroutine. You must call Done with item when you // have finished processing it.

So we'll leak them and/or reuse them perhaps inappropriately. I imagine this could be pretty hard on kube-proxy after a while. Discovered while working on . was suggesting implementing a generic delete-when-parent-is-deleted mechanism, so I may not try to fix this 100% with my current work on .
Generic mechanism proposal was here:
I think it should be enough to check if endpoints match services once just after we initialize, right?
Yeah that should be enough to close this. Although the general mechanism would be nice, too.


Question: nodeports.Name:           nodeports.New, nodepreferavoidpods.Name: nodepreferavoidpods.New, nodeaffinity.Name:        nodeaffinity.New, <ins> podtopologyspread.Name:   podtopologyspread.New, </ins> volumebinding.Name: func(_ *runtime.Unknown, _ framework.FrameworkHandle) (framework.Plugin, error) { return volumebinding.NewFromVolumeBinder(args.VolumeBinder), nil },

What would you like to be : Add a filter plugin that calls into EvenPodsSpread predicate, example Why is this needed: Part of /sig scheduling /help /priority important-soon
/assign
/remove-help
/retitle: [migration phase 1] EvenPodsSpread as filter plugin
/retitle [migration phase 1] EvenPodsSpread as filter plugin
/assign /unassign
May I take over this?
is going to take this over after the offline discussion, and he is familiar with EvenPodsSpread
IC, thanks a lot.


Question: <del> #!/bin/bash # Copyright 2014 The Kubernetes Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Sane defaults for dev environments. The following variables can be easily overriden # by setting each as a ENV variable ahead of time: # KUBE_IMAGE, KUBE_MASTER_FLAVOR, KUBE_NODE_FLAVOR, NUM_NODES, NOVA_NETWORK and SSH_KEY_NAME # Shared KUBE_IMAGE="${KUBE_IMAGE-3eba4fbb-51da-4233-b699-8a4030561add}" # CoreOS (Stable) SSH_KEY_NAME="${SSH_KEY_NAME-id_kubernetes}" NOVA_NETWORK_LABEL="kubernetes-pool-net" NOVA_NETWORK_CIDR="${NOVA_NETWORK-192.168.0.0/24}" INSTANCE_PREFIX="kubernetes" # Master KUBE_MASTER_FLAVOR="${KUBE_MASTER_FLAVOR-general1-1}" MASTER_NAME="${INSTANCE_PREFIX}-master" MASTER_TAG="tags=${INSTANCE_PREFIX}-master" # Node KUBE_NODE_FLAVOR="${KUBE_NODE_FLAVOR-general1-2}" NUM_NODES="${NUM_NODES-4}" NODE_TAG="tags=${INSTANCE_PREFIX}-node" NODE_NAMES=($(eval echo ${INSTANCE_PREFIX}-node-{1..${NUM_NODES}})) KUBE_NETWORK="10.240.0.0/16" SERVICE_CLUSTER_IP_RANGE="10.0.0.0/16"  # formerly PORTAL_NET # Optional: Enable node logging. ENABLE_NODE_LOGGING=false LOGGING_DESTINATION=elasticsearch # Optional: When set to true, Elasticsearch and Kibana will be setup as part of the cluster bring up. ENABLE_CLUSTER_LOGGING=false ELASTICSEARCH_LOGGING_REPLICAS=1 # Optional: Cluster monitoring to setup as part of the cluster bring up: #   none     - No cluster monitoring setup #   influxdb - Heapster, InfluxDB, and Grafana #   google   - Heapster, Google Cloud Monitoring, and Google Cloud Logging ENABLE_CLUSTER_MONITORING="${KUBE_ENABLE_CLUSTER_MONITORING:-influxdb}" # Optional: Install cluster DNS. ENABLE_CLUSTER_DNS="${KUBE_ENABLE_CLUSTER_DNS:-true}" DNS_SERVER_IP="10.0.0.10" DNS_DOMAIN="cluster.local" # Optional: Enable DNS horizontal autoscaler ENABLE_DNS_HORIZONTAL_AUTOSCALER="${KUBE_ENABLE_DNS_HORIZONTAL_AUTOSCALER:-false}" # Optional: Install Kubernetes UI ENABLE_CLUSTER_UI="${KUBE_ENABLE_CLUSTER_UI:-true}" </del>

Rackspace support has not ben updated in a few months. cluster/rackspace/cloud-config/minion-cloud- is using etcd_servers for kubelet and kube-proxy, but support for them has been removed since v0.9, I think. I'll remove it in a few days if no-one makes a counter-offer to get the config working with latest release.
There's been some work on this but it is blocked while we figure out some load-balancer design stuff. , Eric Tune wrote:
Okay, I will remove the link to rackspace until it is fixed (). Leaving this issue open to remind us to delete it in, say, a month, if no progress is made.
cc
See also
I'd prefer to try to get things updated and keep support. However I don't want to continually become stale. every few months either.
Hi all, I want to use kubernetes on my existing rackspace infrastructure but I don't understand if rackspace suppor was removed or not. In Doc, under "custom cloud solution" it's linked but I wasn't able to deploy it neither with install method nor with . With i got a error (I have 16GB or RAM!). thanks in advance, Fx
It looks like no support has happened on Rackspace for a year.
You should not expect it to work, and we should delete rackspace support.
ACK.
Unfortunately I think we'll need to drop support for Rackspace as a deployment target since I can't commit to keeping the deployment scripts up to date. Rackspace support as a cloudprovider should be maintained.
I would be willing to support the Rackspace deployment target, if there is interest to keep it alive.
There would be definitely an interest from our side - sad to see that Rackspace has no interest in maintaining it
No problem, I will take over the maintenance. It will take me a bit of time to get caught up, but I will get it done as soon as possible.
Do you think it would be hard to get the of LBaaS working ?
Sorry it took me a bit to respond. there is a quick update as to where I am. I was busy with some stuff, that prevented me from working on this. But that has been resolved, and should have updates by the end of next week.

However what's causing the delays is the lack of a good API for Rackspace Cloud. I've used the gophercloud, it works but I'm not a fan. Additionally gophercloud doesn't have full API support (one of the things I'm not a fan). This would make it hard to implement LBaaS. I should have a better idea of your request sometime next week.
I have seen this api docu: - is it the same as you saw? I think it should be doable with it.


Question: package kuberuntime import ( <del> "bufio" "bytes" "encoding/json" "errors" "fmt" </del> "io" <del> "math" "os" </del> "time" <del> "github.com/docker/docker/pkg/jsonlog" "github.com/fsnotify/fsnotify" "github.com/golang/glog" </del> "k8s.io/api/core/v1" <del> runtimeapi "k8s.io/kubernetes/pkg/kubelet/apis/cri/v1alpha1/runtime" "k8s.io/kubernetes/pkg/util/tail" </del> <ins> "k8s.io/kubernetes/pkg/kubelet/kuberuntime/logs" </ins> ) <del> // Notice that the current kuberuntime logs implementation doesn't handle // log rotation. // * It will not retrieve logs in rotated log file. // * If log rotation happens when following the log: //   * If the rotation is using create mode, we'll still follow the old file. //   * If the rotation is using copytruncate, we'll be reading at the original position and get nothing. // TODO(random-liu): Support log rotation. // streamType is the type of the stream. type streamType string const ( stderrType streamType = "stderr" stdoutType streamType = "stdout" // timeFormat is the time format used in the log. timeFormat = time.RFC3339Nano // blockSize is the block size used in tail. blockSize = 1024 // stateCheckPeriod is the period to check container state while following // the container log. Kubelet should not keep following the log when the // container is not running. stateCheckPeriod = 5 * time.Second ) var ( // eol is the end-of-line sign in the log. eol = []byte{'n'} // delimiter is the delimiter for timestamp and streamtype in log line. delimiter = []byte{' '} ) // logMessage is the internal log type. type logMessage struct { timestamp time.Time stream    streamType log       []byte } // reset resets the log to nil. func (l *logMessage) reset() { l.timestamp = time.Time{} l.stream = "" l.log = nil } // logOptions is the internal type of all log options. type logOptions struct { tail      int64 bytes     int64 since     time.Time follow    bool timestamp bool } // newLogOptions convert the v1.PodLogOptions to internal logOptions. func newLogOptions(apiOpts *v1.PodLogOptions, now time.Time) *logOptions { opts := &logOptions{ tail:      -1, // -1 by default which means read all logs. bytes:     -1, // -1 by default which means read all logs. follow:    apiOpts.Follow, timestamp: apiOpts.Timestamps, } if apiOpts.TailLines != nil { opts.tail = *apiOpts.TailLines } if apiOpts.LimitBytes != nil { opts.bytes = *apiOpts.LimitBytes } if apiOpts.SinceSeconds != nil { opts.since = now.Add(-time.Duration(*apiOpts.SinceSeconds) * time.Second) } if apiOpts.SinceTime != nil && apiOpts.SinceTime.After(opts.since) { opts.since = apiOpts.SinceTime.Time } return opts } </del> // ReadLogs read the container log and redirect into stdout and stderr. // Note that containerID is only needed when following the log, or else // just pass in empty string "". func (m *kubeGenericRuntimeManager) ReadLogs(path, containerID string, apiOpts *v1.PodLogOptions, stdout, stderr io.Writer) error { <del> f, err := os.Open(path) if err != nil { return fmt.Errorf("failed to open log file %q: %v", path, err) } defer f.Close() </del> // Convert v1.PodLogOptions into internal log options. <del> opts := newLogOptions(apiOpts, time.Now()) // Search start point based on tail line. start, err := tail.FindTailLineStartIndex(f, opts.tail) if err != nil { return fmt.Errorf("failed to tail %d lines of log file %q: %v", opts.tail, path, err) } if _, err := f.Seek(start, os.SEEK_SET); err != nil { return fmt.Errorf("failed to seek %d in log file %q: %v", start, path, err) } // Start parsing the logs. r := bufio.NewReader(f) // Do not create watcher here because it is not needed if `Follow` is false. var watcher *fsnotify.Watcher var parse parseFunc writer := newLogWriter(stdout, stderr, opts) msg := &logMessage{} for { l, err := r.ReadBytes(eol[0]) if err != nil { if err != io.EOF { // This is an real error return fmt.Errorf("failed to read log file %q: %v", path, err) } if !opts.follow { // Return directly when reading to the end if not follow. if len(l) > 0 { glog.Warningf("Incomplete line in log file %q: %q", path, l) } glog.V(2).Infof("Finish parsing log file %q", path) return nil } // Reset seek so that if this is an incomplete line, // it will be read again. if _, err := f.Seek(-int64(len(l)), os.SEEK_CUR); err != nil { return fmt.Errorf("failed to reset seek in log file %q: %v", path, err) } if watcher == nil { // Intialize the watcher if it has not been initialized yet. if watcher, err = fsnotify.NewWatcher(); err != nil { return fmt.Errorf("failed to create fsnotify watcher: %v", err) } defer watcher.Close() if err := watcher.Add(f.Name()); err != nil { return fmt.Errorf("failed to watch file %q: %v", f.Name(), err) } } // Wait until the next log change. if found, err := m.waitLogs(containerID, watcher); !found { return err } continue } if parse == nil { // Intialize the log parsing function. parse, err = getParseFunc(l) if err != nil { return fmt.Errorf("failed to get parse function: %v", err) } } // Parse the log line. msg.reset() if err := parse(l, msg); err != nil { glog.Errorf("Failed with err %v when parsing log for log file %q: %q", err, path, l) continue } // Write the log line into the stream. if err := writer.write(msg); err != nil { if err == errMaximumWrite { glog.V(2).Infof("Finish parsing log file %q, hit bytes limit %d(bytes)", path, opts.bytes) return nil } glog.Errorf("Failed with err %v when writing log for log file %q: %+v", err, path, msg) return err } } } // waitLogs wait for the next log write. It returns a boolean and an error. The boolean // indicates whether a new log is found; the error is error happens during waiting new logs. func (m *kubeGenericRuntimeManager) waitLogs(id string, w *fsnotify.Watcher) (bool, error) { errRetry := 5 for { select { case e := <-w.Events: switch e.Op { case fsnotify.Write: return true, nil default: glog.Errorf("Unexpected fsnotify event: %v, retrying...", e) } case err := <-w.Errors: glog.Errorf("Fsnotify watch error: %v, %d error retries remaining", err, errRetry) if errRetry == 0 { return false, err } errRetry-- case <-time.After(stateCheckPeriod): s, err := m.runtimeService.ContainerStatus(id) if err != nil { return false, err } // Only keep following container log when it is running. if s.State != runtimeapi.ContainerState_CONTAINER_RUNNING { glog.Errorf("Container %q is not running (state=%q)", id, s.State) // Do not return error because it's normal that the container stops // during waiting. return false, nil } } } } // parseFunc is a function parsing one log line to the internal log type. // Notice that the caller must make sure logMessage is not nil. type parseFunc func([]byte, *logMessage) error var parseFuncs []parseFunc = []parseFunc{ parseCRILog,        // CRI log format parse function parseDockerJSONLog, // Docker JSON log format parse function } // parseCRILog parses logs in CRI log format. CRI Log format example: //   2016-10-06T00:17:09.669794202Z stdout log content 1 //   2016-10-06T00:17:09.669794203Z stderr log content 2 func parseCRILog(log []byte, msg *logMessage) error { var err error // Parse timestamp idx := bytes.Index(log, delimiter) if idx < 0 { return fmt.Errorf("timestamp is not found") } msg.timestamp, err = time.Parse(timeFormat, string(log[:idx])) if err != nil { return fmt.Errorf("unexpected timestamp format %q: %v", timeFormat, err) } // Parse stream type log = log[idx+1:] idx = bytes.Index(log, delimiter) if idx < 0 { return fmt.Errorf("stream type is not found") } msg.stream = streamType(log[:idx]) if msg.stream != stdoutType && msg.stream != stderrType { return fmt.Errorf("unexpected stream type %q", msg.stream) } // Get log content msg.log = log[idx+1:] return nil } // parseDockerJSONLog parses logs in Docker JSON log format. Docker JSON log format // example: //   {"log":"content 1","stream":"stdout","time":"2016-10-20T18:39:20.57606443Z"} //   {"log":"content 2","stream":"stderr","time":"2016-10-20T18:39:20.57606444Z"} func parseDockerJSONLog(log []byte, msg *logMessage) error { var l = &jsonlog.JSONLog{} l.Reset() // TODO: JSON decoding is fairly expensive, we should evaluate this. if err := json.Unmarshal(log, l); err != nil { return fmt.Errorf("failed with %v to unmarshal log %q", err, l) } msg.timestamp = l.Created msg.stream = streamType(l.Stream) msg.log = []byte(l.Log) return nil } // getParseFunc returns proper parse function based on the sample log line passed in. func getParseFunc(log []byte) (parseFunc, error) { for _, p := range parseFuncs { if err := p(log, &logMessage{}); err == nil { return p, nil } } return nil, fmt.Errorf("unsupported log format: %q", log) } // logWriter controls the writing into the stream based on the log options. type logWriter struct { stdout io.Writer stderr io.Writer opts   *logOptions remain int64 } // errMaximumWrite is returned when all bytes have been written. var errMaximumWrite = errors.New("maximum write") // errShortWrite is returned when the message is not fully written. var errShortWrite = errors.New("short write") func newLogWriter(stdout io.Writer, stderr io.Writer, opts *logOptions) *logWriter { w := &logWriter{ stdout: stdout, stderr: stderr, opts:   opts, remain: math.MaxInt64, // initialize it as infinity } if opts.bytes >= 0 { w.remain = opts.bytes } return w } </del> <ins> opts := logs.NewLogOptions(apiOpts, time.Now()) </ins> <del> // writeLogs writes logs into stdout, stderr. func (w *logWriter) write(msg *logMessage) error { if msg.timestamp.Before(w.opts.since) { // Skip the line because it's older than since return nil } line := msg.log if w.opts.timestamp { prefix := append([]byte(msg.timestamp.Format(timeFormat)), delimiter[0]) line = append(prefix, line...) } // If the line is longer than the remaining bytes, cut it. if int64(len(line)) > w.remain { line = line[:w.remain] } // Get the proper stream to write to. var stream io.Writer switch msg.stream { case stdoutType: stream = w.stdout case stderrType: stream = w.stderr default: return fmt.Errorf("unexpected stream type %q", msg.stream) } n, err := stream.Write(line) w.remain -= int64(n) if err != nil { return err } // If the line has not been fully written, return errShortWrite if n < len(line) { return errShortWrite } // If there are no more bytes left, return errMaximumWrite if w.remain <= 0 { return errMaximumWrite } return nil </del> <ins> return logs.ReadLogs(path, containerID, opts, m.runtimeService, stdout, stderr) </ins> }

We should make the CRI log parsing code in kubelet to be a library. So that container runtimes and external tools don't need to copy the whole code ( vendor it instead ). Referring
/assign
/sig node
I can help on it.
/assign
GitHub didn't allow me to assign the following users: yanxuean. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks. I have a branch ready for this. Will open a PR later and ping you when it is out.
ok


Question: test/e2e/apimachinery/watch.go: "should be able to start watching from a specific resource version" test/e2e/apimachinery/watch.go: "should be able to restart watching from the last resource version observed by the previous watch" test/e2e/apimachinery/watch.go: "should observe an object deletion if it stops meeting the requirements of the selector" <ins> test/e2e/apimachinery/watch.go: "should receive events on concurrent watches in same order" </ins> test/e2e/apps/daemon_set.go: "should run and stop simple daemon" test/e2e/apps/daemon_set.go: "should run and stop complex daemon" test/e2e/apps/daemon_set.go: "should retry creating failed daemon pods"

We should add a watch test that: a watch a background thread, makes a bunch of random changes the RV from the first item received by the watch. each subsequent item X, open a new watch at the previous RV and verify that the first thing received matches X. Record X's RV for the next iteration. /sig api-machinery
cc
/assign cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
A test for this was via , we just need to promote it to be a conformance test


Question: f := framework.NewDefaultFramework("volumeattachment") <ins> /* Release: v1.30 Testname: VolumeAttachment, lifecycle Description: Creating an initial VolumeAttachment MUST succeed. Reading the VolumeAttachment MUST succeed with with required name retrieved. Patching a VolumeAttachment MUST succeed with its new label found. Listing VolumeAttachment with a labelSelector MUST succeed with a single item retrieved. Deleting a VolumeAttachment MUST succeed and it MUST be confirmed. Creating a second VolumeAttachment MUST succeed. Updating the second VolumentAttachment with a new label MUST succeed with its new label found. Creating a third VolumeAttachment MUST succeed. Updating the third VolumentAttachment with a new label MUST succeed with its new label found. Deleting both VolumeAttachments via deleteCollection MUST succeed and it MUST be confirmed. */ </ins> ginkgo.Describe("Conformance", func() { <del> ginkgo.It("should run through the lifecycle of a VolumeAttachment", func(ctx context.Context) { </del> <ins> framework.ConformanceIt("should run through the lifecycle of a VolumeAttachment", func(ctx context.Context) { </ins> vaClient := f.ClientSet.StorageV1().VolumeAttachments()

[x] two weeks soak end date : 14 Mar 2024 : According to following APIsnoop query, there are six VolumeAttachment endpoints that are untested. Using a number of existing e2e test practices a new has been created to provide future Conformance coverage for the 7 endpoints. The e2e logs for this test are listed below. This query shows the following VolumeAttachment endpoints are hit within a short period of running this e2e test. If a test with these calls gets merged, test coverage will go up by 7 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // An event happens. q.MoveAllToActiveOrBackoffQueue("deleted pod") <ins> q.lock.RLock() </ins> if _, ok, _ := q.podBackoffQ.Get(podInfo); !ok { t.Errorf("pod %v is not in the backoff queue", podID) } <ins> q.lock.RUnlock() </ins> // Check backoff duration. deadline := q.getBackoffTime(podInfo)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-bazel-test Which test(s) are flaking: pkg/scheduler/internal/queue Testgrid link: Reason for failure: Anything else we need to know: /sig scheduling /assign
I just sent a pr for it according to the error message, but I'm not very sure if it will make things all right.
/assign


Question: } // DeletePodWithWaitByName deletes the named and namespaced pod and waits for the pod to be terminated. Resilient to the pod <del> // not existing. Also waits for all owned resources to be deleted. </del> <ins> // not existing. </ins> func DeletePodWithWaitByName(c clientset.Interface, podName, podNamespace string) error { e2elog.Logf("Deleting pod %q in namespace %q", podName, podNamespace) <del> deletionPolicy := metav1.DeletePropagationForeground err := c.CoreV1().Pods(podNamespace).Delete(context.TODO(), podName, metav1.DeleteOptions{ // If the pod is the owner of some resources (like ephemeral inline volumes), // then we want to be sure that those are also gone before we return. // Blocking pod deletion via metav1.DeletePropagationForeground achieves that. PropagationPolicy: &deletionPolicy, }) </del> <ins> err := c.CoreV1().Pods(podNamespace).Delete(context.TODO(), podName, metav1.DeleteOptions{}) </ins> if err != nil { if apierrors.IsNotFound(err) { return nil // assume pod was already deleted

Which jobs are failing: ci-kubernetes-node-kubelet-conformance Which test(s) are failing: Variable Expansion should succeed in writing subpaths in container Variable Expansion should verify that a failing subpath expansion can be modified during the lifecycle of a container Since when has it been failing: Since Fri Jul 10 15:41:46 2020 -0700, when was merged Testgrid link: https://k8s- Reason for failure: Used DeletePropagationForeground option to delete test pod introduced by Anything else we need to know: The test pod stuck in Terminating state until 5m timeout of DeletePodWithWait expired. It happens with empty pod as well. Here is a simplified ginkgo test case that triggers this failure: The pod phase is "Failed" and container reason is "Error" for some reason: Reverting above commit or changing delete option to DeletePropagationBackground should fix this.
/cc
/sig node
/sig storage cc
/assign
I think change just triggered the underlying issue. DeletePropagationForeground option doesn't work as expected it seems.
does the ci-kubernetes-node-kubelet-conformance job run the controller-manager? the kube-controller-manager is responsible for deleting objects with a foreground finalizer on them, but I don't see any kube-controller-manager logs gathered in the artifacts for one of those test failures -
I could reproduce the issue for commit (before 'generic ephemeral inline volumes' PR was merged) + 'use foreground deletion' commit.
yeah, I tend to think switching that e2e helper method for all callers was not a good idea
it looks like node e2e runs only etcd, api server and namespace controller: I don't think it makes sense to run controller-manager just to fix this failure. Would modifying test framework to set DeletePropagationBackground option for e2e node tests be correct solution?
For a quick fix, I would suggest just reverting the test change for now. Long term, yes, it should be configurable per test.
Also do we have plans to make this job a blocking presubmit test so we can find these problems earlier?
I agree, we should revert , both to avoid changing behavior for existing callers and to match how pods are normally deleted
if we need a PR for that I can prepare it.
that would be great, thanks
putting my hat on, probably not?

more hour+ tests that run on every PR are a non-goal
The SIG / OWNERs should enable testgrid alerts on failures, if periodic testing is not sufficient you can use a postsubmit to run whenever we merge code. We're not looking to rely further on presubmit, as is we're running too much there leading to excessive resource usage and inability to merge fixes like this in a timely manner between one thing or the other flaking heavily (and our odds get much worse with each new presubmit)


Question: return status, true, nil } <del> func getPublicIPDomainNameLabel(service *v1.Service) string { </del> <ins> func getPublicIPDomainNameLabel(service *v1.Service) (string, bool) { </ins> if labelName, found := service.Annotations[ServiceAnnotationDNSLabelName]; found { <del> return labelName </del> <ins> return labelName, found </ins> } <del> return "" </del> <ins> return "", false </ins> } // EnsureLoadBalancer creates a new load balancer 'name', or updates the existing one. Returns the status of the balancer

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: After issue is fixed. Cloud provider will automatically reconcile the Azure Public Ip DNS record same as the service annotation. As a result, user finds their DNS record will be deleted if the annotation is not set. Such a change is a block to users who have set the DNS record on their public IP without using this annotation when they upgrade the cluster. Besides, if the public ip is a user's own public ip, it is also not reasonable to delete the DNS record. What you expected to happen: We may need to change the reconcile logic. Maybe only delete the DNS record, when the annotation exists but empty value How to reproduce it (as minimally and precisely as possible): Manually add the DNS label to Azure Public IP resource via Portal Environment: Kubernetes version (use ): 1.15.7 Cloud provider or hardware configuration: Azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cloud-provider
/area provider/azure
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig Azure Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
The fix helm install helm-nginx-ingress stable/nginx-ingress  --namespace ingress-nginx-ns  --set controller.replicaCount=2  --set controller.nodeSelector."beta.kubernetes.io/os"=linux  --set controller.service.annotations."service.beta.kubernetes.io/azure-dns-label-name"=MY-AZURE-DNS  --set defaultBackend.nodeSelector."beta.kubernetes.io/os"=linux


Question: } } <ins> func TestReadRotatedLog(t *testing.T) { tmpDir := t.TempDir() file, err := os.CreateTemp(tmpDir, "logfile") if err != nil { assert.NoErrorf(t, err, "unable to create temp file") } stdoutBuf := &bytes.Buffer{} stderrBuf := &bytes.Buffer{} containerID := "fake-container-id" fakeRuntimeService := &apitesting.FakeRuntimeService{ Containers: map[string]*apitesting.FakeContainer{ containerID: { ContainerStatus: runtimeapi.ContainerStatus{ State: runtimeapi.ContainerState_CONTAINER_RUNNING, }, }, }, } ctx, cancel := context.WithCancel(context.Background()) defer cancel() // Start to follow the container's log. go func(ctx context.Context) { podLogOptions := v1.PodLogOptions{ Follow: true, } opts := NewLogOptions(&podLogOptions, time.Now()) ReadLogs(ctx, file.Name(), containerID, opts, fakeRuntimeService, stdoutBuf, stderrBuf) }(ctx) // log in stdout expectedStdout := "line0nline2nline4nline6nline8n" // log in stderr expectedStderr := "line1nline3nline5nline7nline9n" dir := filepath.Dir(file.Name()) baseName := filepath.Base(file.Name()) // Write 10 lines to log file. // Let ReadLogs start. time.Sleep(50 * time.Millisecond) for line := 0; line < 10; line++ { // Write the first three lines to log file now := time.Now().Format(types.RFC3339NanoLenient) if line%2 == 0 { file.WriteString(fmt.Sprintf( `{"log":"line%dn","stream":"stdout","time":"%s"}`+"n", line, now)) } else { file.WriteString(fmt.Sprintf( `{"log":"line%dn","stream":"stderr","time":"%s"}`+"n", line, now)) } time.Sleep(1 * time.Millisecond) if line == 5 { file.Close() // Pretend to rotate the log. rotatedName := fmt.Sprintf("%s.%s", baseName, time.Now().Format("220060102-150405")) rotatedName = filepath.Join(dir, rotatedName) if err := os.Rename(filepath.Join(dir, baseName), rotatedName); err != nil { assert.NoErrorf(t, err, "failed to rotate log %q to %q", file.Name(), rotatedName) return } newF := filepath.Join(dir, baseName) if file, err = os.Create(newF); err != nil { assert.NoError(t, err, "unable to create new log file") return } time.Sleep(20 * time.Millisecond) } } time.Sleep(20 * time.Millisecond) // Make the function ReadLogs end. fakeRuntimeService.Lock() fakeRuntimeService.Containers[containerID].State = runtimeapi.ContainerState_CONTAINER_EXITED fakeRuntimeService.Unlock() assert.Equal(t, expectedStdout, stdoutBuf.String()) assert.Equal(t, expectedStderr, stderrBuf.String()) } </ins> func TestParseLog(t *testing.T) { timestamp, err := time.Parse(timeFormatIn, "2016-10-20T18:39:20.57606443Z") assert.NoError(t, err)

won't output logs after kubelet rotate log filer of the container. kubectl output logs correctly after the log file is rotated. the config to . So kubelet will rotate the log soon. Build a image with this program a deploy with the previous image. the command No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/area kubelet
/triage accepted /priority important-longterm
/assign since you sent a PR already
Ran into this issue!


Question: return tabbedString(func(out io.Writer) error { w := NewPrefixWriter(out) w.Write(LEVEL_0, "Name:t%sn", pv.Name) <del> printLabelsMultiline(w, "Labels", pv.Labels) printAnnotationsMultiline(w, "Annotations", pv.Annotations) </del> <ins> printLabelsMultiline(w, "Labels", pv.ObjectMeta.Labels) printAnnotationsMultiline(w, "Annotations", pv.ObjectMeta.Annotations) </ins> w.Write(LEVEL_0, "Finalizers:t%vn", pv.ObjectMeta.Finalizers) w.Write(LEVEL_0, "StorageClass:t%sn", helper.GetPersistentVolumeClass(pv)) if pv.ObjectMeta.DeletionTimestamp != nil {


Question: }, expectedErr: "Error from the ConsumeRequestFn", }, <ins> { name: "get logs from multiple requests and ignores the error if the container fails", opts: func(streams genericclioptions.IOStreams) *LogsOptions { mock := &logTestMock{ logsForObjectRequests: map[corev1.ObjectReference]restclient.ResponseWrapper{ { Kind:      "Pod", Name:      "some-pod-error-container", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{err: errors.New("error-container")}, { Kind:      "Pod", Name:      "some-pod-1", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{data: strings.NewReader("test log content from source 1n")}, { Kind:      "Pod", Name:      "some-pod-2", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{data: strings.NewReader("test log content from source 2n")}, }, } o := NewLogsOptions(streams, false) o.LogsForObject = mock.mockLogsForObject o.ConsumeRequestFn = mock.mockConsumeRequest o.IgnoreLogErrors = true return o }, expectedOutSubstrings: []string{ "error-containern", "test log content from source 1n", "test log content from source 2n", }, }, { name: "get logs from multiple requests and an container fails", opts: func(streams genericclioptions.IOStreams) *LogsOptions { mock := &logTestMock{ logsForObjectRequests: map[corev1.ObjectReference]restclient.ResponseWrapper{ { Kind:      "Pod", Name:      "some-pod-error-container", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{err: errors.New("error-container")}, { Kind:      "Pod", Name:      "some-pod", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{data: strings.NewReader("test log content from sourcen")}, }, } o := NewLogsOptions(streams, false) o.LogsForObject = mock.mockLogsForObject o.ConsumeRequestFn = mock.mockConsumeRequest return o }, expectedErr: "error-container", }, { name: "follow logs from multiple requests and ignores the error if the container fails", opts: func(streams genericclioptions.IOStreams) *LogsOptions { mock := &logTestMock{ logsForObjectRequests: map[corev1.ObjectReference]restclient.ResponseWrapper{ { Kind:      "Pod", Name:      "some-pod-error-container", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{err: errors.New("error-container")}, { Kind:      "Pod", Name:      "some-pod-1", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{data: strings.NewReader("test log content from source 1n")}, { Kind:      "Pod", Name:      "some-pod-2", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{data: strings.NewReader("test log content from source 2n")}, }, } o := NewLogsOptions(streams, false) o.LogsForObject = mock.mockLogsForObject o.ConsumeRequestFn = mock.mockConsumeRequest o.IgnoreLogErrors = true o.Follow = true return o }, expectedOutSubstrings: []string{ "error-containern", "test log content from source 1n", "test log content from source 2n", }, }, { name: "follow logs from multiple requests and an container fails", opts: func(streams genericclioptions.IOStreams) *LogsOptions { mock := &logTestMock{ logsForObjectRequests: map[corev1.ObjectReference]restclient.ResponseWrapper{ { Kind:      "Pod", Name:      "some-pod-error-container", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{err: errors.New("error-container")}, { Kind:      "Pod", Name:      "some-pod", FieldPath: "spec.containers{some-container}", }: &responseWrapperMock{data: strings.NewReader("test log content from sourcen")}, }, } o := NewLogsOptions(streams, false) o.LogsForObject = mock.mockLogsForObject o.ConsumeRequestFn = mock.mockConsumeRequest o.Follow = true return o }, expectedErr: "error-container", }, </ins> } for _, test := range tests { t.Run(test.name, func(t *testing.T) {

What happened: Running on a pod with two containers, of which one has restarted but the other hasn't results in a undesired behavior. I get an inconsistent outcome, sometimes just a report of , other times I get the other containers logs followed by the error. What you expected to happen: I expect to get to see the full response of all containers, even though one container without any --previous logs throwed an error. How to reproduce it (as minimally and precisely as possible): The output can look like this... Anything else we need to know?: The flag by itself ("If watching / following pod logs, allow for any errors that occur to be non-fatal") doesn't influence this, but a workaround is to use to be able to see the full output. Kubernetes version (use ): Cloud provider or hardware configuration: GKE OS (e.g: ): Ubuntu 20.04 Kernel (e.g. ): 5.4.0-58-generic
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
do you have an example yaml file to create these containers?
yikes I feared I'd be asked this question :) Well, I provided a minimal example in the original post.
Thank you very much. I'll take a look.
Hmm... I can replicate this issue with the example you provided. Very explanatory instructions! I'll investigate tomorrow.
I used the example to reproduce this issue. Created pod through above. The state of pod is Running. I guess the reason is that none of the containers in pod have Terminated state.Only when one of containers is Terminated. The param --previous will return the result normally.
looking at these lines... ...I discovered that adding the option significantly increases the frequency that the "bash" container's logs are printed (the numbers are how many times was printed out of 100 runs):
Can i work on this issue?
/assign
Go ahead I'd love for you too. Hopefully my notes help!


Question: // no old objects are left in storage), and then remove the rest of the // versions from this list. // Versions may not be removed from `spec.versions` while they exist in this list. <ins> // +optional </ins> repeated string storedVersions = 3; }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, the field on must be explicitly set to their zero-values due to the fact that they are not tagged . Ref: Why is this needed: If using a system that automatically compares the provided configuration to the current, it will produce a diff on the updated status after creation. In addition, making sure to explicitly set to zero-values is not the best user experience. I am happy to provide the fix if this change is approved! /cc /sig api-machinery /area custom-resources
marking them as seems ok, so that client-side validation doesn't complain if they are absent when submitting manifests, but they shouldn't need to be set to omitempty (and at this point, we likely wouldn't change the serialization) Those fields are changed by controllers after creation, so expecting the live API object to exactly match the status is not going to go well anyway.
Good point! I’ll open a fix for this. Yep! The specific system in question ignores fields that are not set at creation time by default when doing comparison, so if status fields are not included then it will not report a diff.
/assign
I opened a fix for this but CC'ed both you and Let me know if any additional effort is required. Thanks!


Question: return nil, framework.NewStatus(framework.Skip) } <del> s.availableOnNodes = make([]*nodeaffinity.NodeSelector, len(claims)) </del> <ins> s.informationsForClaim = make([]informationForClaim, len(claims)) </ins> for index, claim := range claims { if claim.Spec.AllocationMode == resourcev1alpha2.AllocationModeImmediate && claim.Status.Allocation == nil {

The node selectors gets parsed by in for each node. That work can be done once in . Better performance.
/sig node /sig scheduling /triage accepted /priority backlog
I can try this one. /assign


Question: podUID := string(pod.UID) contName := container.Name needsReAllocate := false <del> for k := range container.Resources.Limits { </del> <ins> for k, v := range container.Resources.Limits { </ins> resource := string(k) <del> if !m.isDevicePluginResource(resource) { </del> <ins> if !m.isDevicePluginResource(resource) || v.Value() == 0 { </ins> continue } err := m.callPreStartContainerIfNeeded(podUID, contName, resource)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I register a device plugin and set the option in true, then when I create a pod and specify this resource as 0 in yaml, an error will be reported. Examples are as follows. The request to register the device pluginA pod yaml with the resource 0kubelet return error What you expected to happen: I think when the kubelet calls , the resource value should be checked. If it is 0, the step should be skipped to avoid errors. How to reproduce it (as minimally and precisely as possible): Register a device plugin which set option is true Create a pod with the above resource 0 Anything else we need to know?: In fact, when the amount of requested resources is 0, devicemanager will get the number of allocated devices as nil, resulting in the internal cache not storing the application resource information corresponding to the container. But if is set to true at the same time, then devicemanager will check podDevices cache. The two processing steps contradict each other so it will return an error.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


Question: assert.EqualValues(t, endpoint.TargetRef, &v1.ObjectReference{Kind: "Pod", Namespace: ns, Name: pod1.Name}) } <ins> func TestSyncServiceEndpointSlicePendingDeletion(t *testing.T) { client, esController := newController([]string{"node-1"}, time.Duration(0)) ns := metav1.NamespaceDefault serviceName := "testing-1" service := createService(t, esController, ns, serviceName) err := esController.syncService(fmt.Sprintf("%s/%s", ns, serviceName)) assert.Nil(t, err, "Expected no error syncing service") gvk := schema.GroupVersionKind{Version: "v1", Kind: "Service"} ownerRef := metav1.NewControllerRef(service, gvk) deletedTs := metav1.Now() endpointSlice := &discovery.EndpointSlice{ ObjectMeta: metav1.ObjectMeta{ Name:            "epSlice-1", Namespace:       ns, OwnerReferences: []metav1.OwnerReference{*ownerRef}, Labels: map[string]string{ discovery.LabelServiceName: serviceName, discovery.LabelManagedBy:   controllerName, }, DeletionTimestamp: &deletedTs, }, AddressType: discovery.AddressTypeIPv4, } err = esController.endpointSliceStore.Add(endpointSlice) if err != nil { t.Fatalf("Expected no error adding EndpointSlice: %v", err) } _, err = client.DiscoveryV1().EndpointSlices(ns).Create(context.TODO(), endpointSlice, metav1.CreateOptions{}) if err != nil { t.Fatalf("Expected no error creating EndpointSlice: %v", err) } numActionsBefore := len(client.Actions()) err = esController.syncService(fmt.Sprintf("%s/%s", ns, serviceName)) assert.Nil(t, err, "Expected no error syncing service") // The EndpointSlice marked for deletion should be ignored by the controller, and thus // should not result in more than one action from the client (an update to the non-terminating // EndpointSlice removing the trigger time annotation.) if len(client.Actions()) != numActionsBefore+1 { t.Errorf("Expected 1 more action, got %d", len(client.Actions())-numActionsBefore) } } </ins> // Ensure SyncService correctly selects and labels EndpointSlices. func TestSyncServiceEndpointSliceLabelSelection(t *testing.T) { client, esController := newController([]string{"node-1"}, time.Duration(0))

As discussed in , Endpoint(Slice) controllers can get stuck if an Endpoints or EndpointSlice resource is stuck in a terminating state. I don't think there's much we can do for Endpoints here since the same name is reused, but EndpointSlice controllers can ignore EndpointSlices in a terminating state. See No response All supported versions Any Any Any Any Any
/sig network
/triage accepted
deleting a service with a finalizer doesn't affect the endpointslice, it doesn't change its status to termination or unready Repro: finalizer to the service and delete it, it keeps waiting until the finalizer is removed with the deletion timestamp set. the endpoint slice associated Do you mean that when the endpoint slice controller detects that the Service is being deleted should switch the endpoint slice status to Terminating? cc:
I don't think we need to do anything special if the Service is waiting to be deleted. There are many legitimate reasons for this to happen, including waiting for infrastructure to be removed. My main concern with a finalizer was that one on an EndpointSlice could cause the controller to get stuck on a Service. We don't have the same risk with a finalizer on a Service - I don't think that would cause the controller to get stuck in any way.
i was unable to repro the issue on my machine: I was monitoring the logs of , and the endpointslice controller never complained about , when I deleted the service. Though, I was able to get the controller to log this error under a different scenario. If we don't explicitly remove the finalizer from the endpointslice, and instead re-create the pod and service like: From what I can understand, because the endpointslice with the finalizer is owned by a service with a different UID than the new endpointslice, it is marked as expected for deletion . The endpointslice controllers calls , which returns true for the endpointslice with the finalizer, because of the above reason. I might be wrong anywhere here, so please let me know if that is the case. As for further action, I'm not sure what can be done here? We could check the endpointslice for finalizers and then skip them . Keen to hear your thoughts about this.


Question: github.com/boltdb/bolt v1.3.1 // indirect github.com/clusterhq/flocker-go v0.0.0-20160920122132-2b8b7259d313 github.com/container-storage-interface/spec v1.3.0 <del> github.com/containernetworking/cni v0.8.0 </del> <ins> github.com/containernetworking/cni v0.8.1 </ins> github.com/coredns/corefile-migration v1.0.11 github.com/coreos/go-oidc v2.1.0+incompatible github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e

is identified with with severity rating as high. This is found using snyk scan while running against k/k master We're not using directly however, it gets tagged for vulnerability scans as we use . The fix is already present in and we can bump the dep. Bump dep to which has the mentioned CVE fix. and check Kubernetes version (use ): n/a Cloud provider or hardware configuration: any OS (e.g: ): n/a Kernel (e.g. ): n/a Install tools: n/a Network plugin and version (if this is a network-related bug): n/a Others: n/a /sig architecture /area code-organization for the dependency
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: deps = [ "//staging/src/k8s.io/apiserver/pkg/authentication/authenticator:go_default_library", "//staging/src/k8s.io/apiserver/pkg/authentication/user:go_default_library", <ins> "//staging/src/k8s.io/apiserver/pkg/endpoints/request:go_default_library", </ins> ], )

Semantics are going to vary. We need to cross off these authenticators: [x] if ctx has an audience limitation, verify the token audience and echo in the response [x] take implicit audiences at WebhookTokenAuthenticator construction time if ctx is audience-limited, add into the tokenreview object if the tokenreview returns with an audience set that matches the ctx, copy into the response and return success if the tokenreview returns without an audience set, ensure the ctx audiences intersect with the implicit audiences taken at WebhookTokenAuthenticator construction time, and pick one to add into the response [x] echo audience in response if present in ctx [x] echo audience in response if present in ctx [x] check that the the order of (audience in the context) ∩ ( (implicit api audiences) ∪ (clientids) ) 0 Take implicit audiences at Authenticator construction time, and if ctx is audience-limited, ensure it intersects with the implicit audiences, and echo an audience in the response. [x] [x] [x] [x] [x] [x] Do nothing? [x] [x] [x] [x] [x] [x] [x] note: we already keyed the cache with the audience. [x]
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if err != nil { return "", err } <del> if volume.Status != VolumeAvailableStatus { errmsg := fmt.Sprintf("volume %s status is %s, not %s, can not be attached to instance %s.", volume.Name, volume.Status, VolumeAvailableStatus, instanceID) glog.Errorf(errmsg) return "", errors.New(errmsg) } </del> cClient, err := os.NewComputeV2() if err != nil { return "", err

When detaching disk, if the volume status is already available, detach should return true instead of error.
/sig openstack
Related to


Question: command := args[0] baseCommand := path.Base(command) serverName := baseCommand <ins> args = args[1:] </ins> if serverName == hk.Name { <del> args = args[1:] </del> baseFlags := hk.Flags() baseFlags.SetInterspersed(false) // Only parse flags up to the next real command

When trying to use kubectl from hyperkube I'm met with "unknown flag" errors. It looks as if the flags aren't being passed through to kubectl but hyperkube is trying to handle them itself (and subsequently giving an error when you give it flags it doesn't understand). You can reproduce simply by running: And you'll see the error:
I have a fix up in PR .


Question: testRSLifeCycle(f) }) <del> ginkgo.It("should list and delete a collection of ReplicaSets", func() { </del> <ins> /* Release: v1.22 Testname: ReplicaSet, list and delete a collection of ReplicaSets Description: When a ReplicaSet is created it MUST succeed. It MUST succeed when listing ReplicaSets via a label selector. It MUST succeed when deleting the ReplicaSet via deleteCollection. */ framework.ConformanceIt("should list and delete a collection of ReplicaSets", func() { </ins> listRSDeleteCollection(f) })

[X] test approval issue : [x] test pr : [x] two weeks soak end date : 2 June 2021 [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints which are untested. Create a ReplicaSet List all Replicasets DeleteCollection of ReplicaSets Varify the ReplicaSets have been deleted Using an existing conformance test as a template for a which validates that two new endpoints are hit. example endpoint | useragent --------------------------------------------|
createAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets deleteAppsV1CollectionNamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets listAppsV1ReplicaSetForAllNamespaces | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets readAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets (4 rows) If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
/triage accepted For Release 1.22
/assign
Dear When I run It only returns not listAppsV1ReplicaSetForAllNamespaces. What does this mean? What should I do? Regarding What should I see after implementing the tests? All 4 rows? Right now I see 0 rows when running it (without editing code) Lastly, could you tell me which parts need to be addressed in the ? Right now, it looks like the template does exactly what you want to write for this test.
This issue is created to show a planned PR to the conformance meeting. It is just a proof of concept before creating the conformance PR. What we do: I create a local cluster and create a test. In the cluster, I have running to see what user agents are hit by the test. Before I run the test I delete all cluster logs.

Then I run the test. This is my test: Once the test pass I run the APISnoop query to see which user-agents are hit. I in this case 4 API's are hit, but I am only interested in and as they are the ones yet.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: framework.Logf("Cluster is running on COS. Proceeding with test") if f.BaseName == "device-plugin-gpus" { <del> dsYamlUrl = "https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/master/daemonset.yaml" </del> <ins> dsYamlUrl = "https://raw.githubusercontent.com/GoogleCloudPlatform/container-engine-accelerators/k8s-1.9/daemonset.yaml" </ins> gpuResourceName = framework.NVIDIAGPUResourceName podCreationFunc = makeCudaAdditionDevicePluginTestPod } else {

This issue is here to satisfy PR requirements of issue association. A lot of times we need to do version bumps for minor bug fixes that don't deserve a separate issue in the kubernetes/kubernetes repo. /kind bug /sig node
/lifecycle frozen
all the linked PRs are emrged and no activity since more than 3 years /close
Closing this issue. <detailsIn response to : all the linked PRs are emrged and no activity since more than 3 years /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Testname: Pod with the deleted RuntimeClass is rejected. Description: Pod requesting the deleted RuntimeClass must be rejected. */ <del> ginkgo.It("should reject a Pod requesting a deleted RuntimeClass [NodeConformance]", func() { </del> <ins> framework.ConformanceIt("should reject a Pod requesting a deleted RuntimeClass [NodeConformance]", func() { </ins> rcName := createRuntimeClass(f, "delete-me", "runc", nil) rcClient := f.ClientSet.NodeV1().RuntimeClasses()

Per the comment in this PR conformance tests will regress because of a newly Runtime Class GA v1 API is not covered with tests. Full comment: Full comment: /assign More links:
/sig node /triage accepted
KEP:
New to conformance tests. So the idea is to mark tests (or create new ones) that would call each of these methods at lease once? So if this test will be marked as conformance, it will cover and . Is there any additional requirements for the conformance tests? Maybe with the number of side effects of that test or a special execution environment?
Yes. You can make a single test that runs objects through CRUD operations. For the deletecollection one, I'd suggest labeling the objects under test and scoping the deletecollection to your test label to avoid breaking the cluster you're running on. See as an example
Great, will be working on it! Also just chatted with about it.
Just to get clarity on this - is this considered a blocker for Confirming bc grad criteria for it says "No release blocking feedback for API and functionality" should these tests be required to consider this stable? As a note: Code Freeze means that required tests must also be finished:
Here's possibly an easier link to track: !
This seems like something that should be tested generically, or at the very least have a table-driven test that covers all (or most) GA APIs.
! Just a quick note to show that currently the new tests are hitting all endpoints. I'd go ahead and create the Promotion PR in order to get any necessary feedback from #sig-arch
/reopen re: I disagree with the idea that a single test covering CRUD operations means RuntimeClass is now conformance tested. Generic CRUD on a resource's API is not sufficient to describe the intended behavior of the resource. That kind of test is at best verifying that api machinery is working. A vendor would able to pass this test, without implementing any of the behaviors runtimeclass enables. What does do? What kinds of behaviors are visible to an end-user? Are any of those tests in Conformance?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I'm not familiar enough with runtimeclass to know whether these would be a good start:
Thank you for reopening. I agree with this in principle. But I really would appreciate some guidance here. Scheduling tests that takes runtime handler into account are great. However they don't hit any endpoints. I asked and I'm not clear whether those needs to be covered as they are not touching an API. As for actual scheduling, one limitation is posted . Which makes some other tests impossible to include to conformance. I next week, but there may not be all the right people there as it's a Thanksgiving short week. Do you have a suggestion?
I agree that exercising specific CRI implementations is not the place of Conformance testing. Conformance testing is intended to exercise end-user visible behaviors of a Kubernetes cluster, which basically means whatever you can see and do at the Control Plane API level (ref: ) The challenge we face with Conformance testing is that we don't have an easy way to enumerate and measure coverage of behaviors. We experimented with listing behaviors in .yaml so that humans and machines could populate and approve them independent of test implementation, but ultimately abandoned that effort due to its complexity (ref: ) So we use API coverage as an easily machine-measurable/enforceable proxy. Noticing the coverage was at 0% should have been a sign that we should look at the existing test coverage of RuntimeClass and decide which tests were eligible for promotion to Conformance, and whether any new tests needed to be written. In theory a solid test plan for a KEP would include a list of behaviors that need to be tested. That's not immediately clear to me from the KEP for RuntimeClass (ref: ) Based on my read of the KEP, some valid tests might be: tests that exercise the runtimeclass admission controller creating pods with a non-existent runtimeclass creating pods with an existing runtimeclass creating pods with scheduling rules that would clash with runtimeclass scheduling rules creating pods with scheduling rules that would merge with runtimeclass scheduling rules Tests that would be disallowed by conformance requirements because they would require pre-installation/configuration of the cluster under test: running pods with a runtimeclass (unless there's some way to create a functional "dummy" runtimeclass without touching kubelets) running pods with a runtimeclass with a specific handler such as containerd, crio, etc.

and expecting them to run
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
sorry for the delay /remove-lifecycle stale
/area conformance
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: import ( "errors" <ins> "net/http" </ins> "testing" corev1 "k8s.io/api/core/v1" <ins> apierrors "k8s.io/apimachinery/pkg/api/errors" </ins> "k8s.io/apimachinery/pkg/api/meta" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema"

What happened: When sending a request to the server side apply endpoint (), if the apiVersion provided in the object is different from the apiVersion in the url, it returns a 500 error, like: What you expected to happen: The request should fail if the apiVersion provided in the object is different from the apiVersion in the url, but it should fail before trying to merge, and be a 4xx error. Probably a bad request error. Anything else we need to know?: The check should probably be done somewhere near this . The issue, though, is that the patch is given as a byte array, and needs to be deserialized into an object to access the apiVersion field. How to reproduce it (as minimally and precisely as possible): With a kube-apiserver built from head, running with :
/kind bug /sig api-machinery /wg apply /priority important-longterm /assign


Question: apiVersion: v1 kind: Pod metadata: <del> name: rescheduler-v0.2.2 </del> <ins> name: rescheduler-v0.3.0 </ins> namespace: kube-system labels: k8s-app: rescheduler <del> version: v0.2.2 </del> <ins> version: v0.3.0 </ins> kubernetes.io/cluster-service: "true" kubernetes.io/name: "Rescheduler" spec: hostNetwork: true containers: <del> - image: gcr.io/google-containers/rescheduler:v0.2.2 </del> <ins> - image: gcr.io/google-containers/rescheduler:v0.3.0 </ins> name: rescheduler volumeMounts: - mountPath: /var/log/rescheduler.log

https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite} Previous issues for this test:
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
https://k8s- Failed: [] Rescheduler [Serial] should ensure that critical pod is scheduled in case there is no resources available {Kubernetes e2e suite}
This test always failed in 1.5 upgrade builds for the similar reason as . In previous version it will try to retrieve the ReplicationController of kube-dns during this test(). But since kube-dns started to use Deployment but not RC and leads to this failure in upgrade builds.
This is not a sig/apps issue, relabeling to sig/scheduling
I'd like to follow up for this flaky test :).
This test has been failing consistently since Saturday, which may indicate a release-blocking regression. Will you be able to prioritize investigating the problem?
sure
Check the log at re-scheduler evict pods at ; but did not get scheduler's event. At , the e2e test report error and start to clear up test environment.
And after checking the log, there're only 46 of 60 pods are scheduled. Created a PR that reduced the pods number to see what will happen.
/cc , , for comments.
I have been able to reproduce this, but I still don't have a good answer as why this is happening. I will keep looking at the code and will update here if I find anything.
Can you share some logs? I can not run it locally/or gce because of network; just check the log on the server :).
In the latest run of the test that has failed, I see that scheduler attempts to schedule the critical pod after rescheduler evicts other pods to create room, but the critical pod still does not fit on the nodes: scheduler log (look for "-2n3g2"): rescheduler log: (I will look at the test code later to see what's going on)
Wow, yes, similar conclusion yesterday ().

If you can reproduce it, maybe we can add some instrument/debug log to see in scheduler; that'll be helpful :).
cc/
please correct me if I'm wrong but tolerations/taints is already migrated to fields in HEAD. This means that to fix this issue we need to migrate rescheduler to use fields. I'll do it very soon.
Do you think this issue indicates a regression that should block 1.6, which would require that a fix be available asap? Or is it a test-only issue that can be moved to the 1.6.1 or 1.7 milestone?
the former one. See
yes taints and tolerations are already moved to api fields, and the related PRs are already merged for 1.6. Also this looks duplicate of , I think we should close one.
+1
was closed in favor of this issue, since we expect the bot to re-open this
Status: is working on a fix


Question: apiVersion: v1 kind: ReplicationController metadata: <del> name: kube-dns-v17 </del> <ins> name: kube-dns-v17.1 </ins> namespace: kube-system labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: replicas: {{ pillar['dns_replicas'] }} selector: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> template: metadata: labels: k8s-app: kube-dns <del> version: v17 </del> <ins> version: v17.1 </ins> kubernetes.io/cluster-service: "true" spec: containers:

Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?


Question: } <ins> func TestDescribeNode(t *testing.T) { fake := fake.NewSimpleClientset(&api.Node{ ObjectMeta: metav1.ObjectMeta{ Name:      "bar", Namespace: "foo", }, Spec: api.NodeSpec{ Unschedulable: true, }, }) c := &describeClient{T: t, Namespace: "foo", Interface: fake} d := NodeDescriber{c} out, err := d.Describe("foo", "bar", printers.DescriberSettings{ShowEvents: true}) if err != nil { t.Errorf("unexpected error: %v", err) } expectedOut := []string{"Unschedulable", "true"} for _, expected := range expectedOut { if !strings.Contains(out, expected) { t.Errorf("expected to find %q in output: %q", expected, out) } } } </ins> // boolPtr returns a pointer to a bool func boolPtr(b bool) *bool { o := b

Once that event expires I'm fairly certain I would have no idea that the node is cordoned.
can we add one line when , wdyt?
/assign
We should probably reflect the actual field, rather than call it Cordoned. We try to use (as much as possible) the same terminology in the UI as in the API. Unschedulable would probably be sufficient
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: Resources: v1.ResourceRequirements{ Limits: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(100, resource.DecimalSI), <del> v1.ResourceMemory: *resource.NewQuantity(30*1024*1024, resource.DecimalSI), </del> <ins> v1.ResourceMemory: *resource.NewQuantity(50*1024*1024, resource.DecimalSI), </ins> }, }, },

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig testing What happened: Tests from are spawning pods with memory limits (30 MB). Running the tests using Windows nodes will cause the Windows nodes to end up in NotReady state. This happens because the Docker service couldn't start that container (because of the low memory limit) and it hangs. This is a known issue. [1] Increasing the memory limit to a more reasonable value (50 MB) solves the issue. [1] What you expected to happen: The Docker service shouldn't hang just because of the low memory limit imposed by a test. How to reproduce it (as minimally and precisely as possible): a Kubernetes deployment with Windows nodes (Windows Server 1803) the tests, or just spawn the following pod: the node status: the Windows node, check Docker's responsiveness by issuing a docker CLI command (`). Observe that the CLI will not receive any answer and it will hang. Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.1 Cloud provider or hardware configuration: N/A OS (e.g. from /etc/os-release): Windows nodes (Windows Server 1803) Kernel (e.g. ): N/A Install tools: N/A Others:
/sig windows
This was actually addressed by , but the PR message had the wrong Issue ID. The already merged, so this issue can be closed.


Question: kube::util::require-jq kube::golang::setup_env <ins> kube::etcd::install </ins> make -C "${KUBE_ROOT}" WHAT=cmd/kube-apiserver

we just log that the user should install etcd, which is impractical in an ephemeral container we actually already call installing etcd in the verify script, but we don't export it into the current PATH for the process so it doesn't work ... ref: /sig api-machinery /assign /kind bug
/triage accepted
for one possible fix
/sig etcd


Question: release_filegroup( name = "debs", srcs = [ <del> ":cloud-controller-manager.deb", </del> ":cri-tools.deb", ":kubeadm.deb", ":kubectl.deb",

Is this a BUG REPORT or FEATURE REQUEST? We should stop including the deb for CCM as part of Kubernetes releases. Given we almost always see cloud controller manager run within a container (as recommended). This is consistent with other controllers. /kind bug /sig release /sig cluster-lifecycle /sig cloud-provider cc
Resolved in :tada:


Question: message = strings.TrimSpace(string(body)) } retryAfter, _ := retryAfterSeconds(resp) <del> return errors.NewGenericServerResponse(resp.StatusCode, req.Method, r.resource, r.resourceName, message, retryAfter) </del> <ins> return errors.NewGenericServerResponse(resp.StatusCode, req.Method, r.resource, r.resourceName, message, retryAfter, true) </ins> } // isTextResponse returns true if the response appears to be a textual media type.

If go-restful detects a particular method is not allowed, it will return this error: That is preventing useful methods that resthandler would generate from making it to the client. Expected response:
triggered with
/cc
/cc
Is the right fix to update the client to generate the message versus changing the server to allow the method and report the error? My bias is to improve the client and not change the server as it makes reading our swagger doc easier to not see a bunch of verb unsupported entries. Sent from my iPhone
My bias is also towards not changing the server to keep swagger spec correct. By changing the client, do you mean changing kubectl? can you please clarify how is the json useful to client? It does not contain any useful extra information. On Feb 11, 2015 7:28 PM, "Derek Carr" wrote:
Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to. Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect.
This was an unintended side effect, but I think the old behavior was an unexpected luxury? I agree from the end-user perspective, the old message was nice, and it's great that the client was politely told that a given verb was not supported on the resource, but I don't agree that a server should appear to accept a method to only report that its not allowed. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:04:41 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () Returning an API response would allow clients that expect API status to get nice messages for free, instead of spitting this out: We can certainly put the burden of interpreting 405 errors on all clients, just seemed nice not to.

Also, since the resthandler code expected to be returning API status responses when the method wasn't supported, I wanted to make sure having go-restful intercept and return 405's instead wasn't left as an unintended and unconsidered side-effect. Reply to this email directly or view it on GitHub:
If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error?
I think so, this is only going to get worse when we support specific types of verbs on specific resources. In practice, it seems fine for kubectl to give a better error specific to the method not allowed response. ----- Original Message ----- From: "Jordan Liggitt" To: "GoogleCloudPlatform/kubernetes" Cc: "Derek Carr" Sent: Thursday, February 12, 2015 1:26:08 PM Subject: Re: [kubernetes] go-restful is emitting less helpful errors than resthandler () If we want to leave go-restful returning that as an intended and considered side-effect, that's fine. Is the consensus that this issue should change to update kubectl to show a nicer message for that error? Reply to this email directly or view it on GitHub:
We could consider submitting a PR to go-restful to allow us to provide an error-generation function or interface implementation.
Yeah, that seems reasonable.
Opened
Copying text from : Some errors return as json and some as plain text. This causes overhead on client side when parsing the errors since need to recheck content type for every error in order not to parse plain text as json (some libraries fail on that). The errors message format should be standardized. Examples: The text/plain errors are from go-restful. cc
+1, thanks
The status objects are always of the default API version, however, rather than the version requested, because the request object isn't available to the error handler.
Rather than: we need:
Can we define an error handler on a web service?
At the moment, it's at the container level:
David and I talked about keeping the API lookup mechanism so we could use that in the error handler (apirequestinfo). We still need it for some use cases (proxies) so it could be used here to identify the version and proposed resource if matched.


Question: AzureAuthConfig: auth.AzureAuthConfig{ Cloud: "AZUREPUBLICCLOUD", }, <ins> CloudConfigType: cloudConfigTypeFile, </ins> } <del> az.Config.CloudConfigType = cloudConfigTypeFile </del> err = az.InitializeCloudFromConfig(&config, false) expectedErr = fmt.Errorf("useInstanceMetadata must be enabled without Azure credentials") assert.Equal(t, expectedErr, err)

What happened: While trying to setup a outside of the cluster, I had a panic as follow: What you expected to happen: Produce an error message as my configuration lacks some options. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.18.4 Cloud provider or hardware configuration: Azure Others: --cloud-config= The panics come from because we exit as we're not in this case as the is just defaulted-initialised by the caller to here: and effectively assigned afterwards
/sig cloud-provider
/area provider/azure
cc
thanks for reporting the issue and filing the PR to fix it


Question: apps "k8s.io/api/apps/v1" "k8s.io/api/core/v1" policy "k8s.io/api/policy/v1beta1" <ins> apiequality "k8s.io/apimachinery/pkg/api/equality" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" "k8s.io/apimachinery/pkg/util/uuid"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: There are too many pdb update operations compared with other resources. ! That is, there are many "Put" operations for pdb periodically in kube-apiserver log. In my view, the amount of pdb operations is connected with disruption controller's resyncPeriod and pdb's quantity in cluster. For example: disruption-controlle's default resyncPeriod is 30s, and there are 100 pdbs in cluster. We can find that the update operation times per minute is about 200. The data is reasonable, explanation is as follows: According to disruption-controlle's design, there is one resync every 30s. One resync means to call updatePdbStatus function for every pdb, so there are 200 times updatePdbStatus. Function updatePdbStatus judges whether the pdb is changed. If changed, call apiserver to update the pdb' status. When no eviction happens, disruptedPods is empty. In this situation, disruptedPods is an empty map, but pdb.Status.DisruptedPods is nil, empty map is not equal to nil. So reflect.DeepEqual(pdb.Status.DisruptedPods, disruptedPods) returns false. The pdb is thought to has be changed, then apiserver is called to update the pdb' status. In fact, there is no change for the pdb when disruptedPods is an empty map and pdb.Status.DisruptedPods is nil. What you expected to happen: There shouldn't be so many pdb update operations. That is, when pdb has no change, apiserver shouldn't be called to update its status. How to reproduce it (as minimally and precisely as possible): Create some pdbs in cluster, you will see many "Put" operations for pdb periodically in kube-apiserver log. Anything else we need to know?: Environment: kubernetes v1.13.2 Fix suggestions: The bug is caused by the mixed usage of empty and nil for disruptedPods, the type of which is map[string]metav1.Time.

There are two fix suggestions for it: one of empty map and nil for disruptedPods, use it consistently and refuse to mix empty map and nil. For example, when disruptedPods is empty or nil, save it as an empty map in etcd. Then every client gets pdb from etcd, the pdb.Status.DisruptedPods is an empty map. get pdb for etcd, assign an empty map to it if pdb.Status.DisruptedPods is nil. Maybe the mixed usage of empty and nil for map is a common problem in k8s design, not just for disruptedPods. I will propose a work around solution for the problem. If the solution has any problem, please let me know and I can modify the code.
/sig apps
/assign Morten can you PTAL and the open PR. I reviewed as well.
ref


Question: jobColumnDefinitions := []metav1.TableColumnDefinition{ {Name: "Name", Type: "string", Format: "name", Description: metav1.ObjectMeta{}.SwaggerDoc()["name"]}, <ins> {Name: "Status", Type: "string", Description: "Status of the job."}, </ins> {Name: "Completions", Type: "string", Description: batchv1.JobStatus{}.SwaggerDoc()["succeeded"]}, {Name: "Duration", Type: "string", Description: "Time required to complete the job."}, {Name: "Age", Type: "string", Description: metav1.ObjectMeta{}.SwaggerDoc()["creationTimestamp"]},

Suspend is a GA feature and it is big component of Kueue. It would be nice to display if jobs are suspended when doing . A new column that displays if the job is suspended would be helpful. Improve UX around suspend. Its a bit difficult to know why the job has not created pods unless you know about the suspend field.
/sig cli /wg batch /good-first-issue /help-wanted
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig cli /wg batch /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc Just something I noticed as I was testing some job in JobSet. It would be nice to see which jobs are suspended when doing . Hope its not a controversial change.
cc who worked on
Not controversial at all IMO /triage accepted
/assign
Any suggestions on where to place this column? I'm thinking between completions and durations. Or, should this be displayed only when the output is set to wide?
I would put it in the default view as well. Probably before completions
Addressed in the PR:


Question: delete(s.Properties, "apiVersion") delete(s.Properties, "kind") delete(s.Extensions, "x-kubernetes-group-version-kind") <ins> delete(s.Extensions, "x-kubernetes-selectable-fields") </ins> } func verifyKubectlExplain(ns, name, pattern string) error {

<img width="1480" alt="image" src="" /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig api-machinery
cc in case you know of something that just merged
Last good run : First bad run : k8s version stamp: GOOD: "v1.30.0-alpha.3.56+" BAD : "v1.30.0-alpha.3.62+" Here's the comparison of what went in: We have a winner :) - Will leave this to !
/assign
Thanks I've opened and kicked off the alpha tests on that PR to ensure the fix sticks.
These tests appear to be passing in latest runs: (although NPD is failing)
yea, for the NPD failure i had a cherry pick to containerd 1.7 here -
/triage accepted


Question: }, nil } } <ins> // FakeReservePlugin is a test reserve plugin. type FakeReservePlugin struct { Status *framework.Status } // Name returns name of the plugin. func (pl *FakeReservePlugin) Name() string { return "FakeReserve" } // Reserve invoked at the Reserve extension point. func (pl *FakeReservePlugin) Reserve(_ context.Context, _ *framework.CycleState, _ *v1.Pod, _ string) *framework.Status { return pl.Status } // Unreserve invoked at the Unreserve extension point. func (pl *FakeReservePlugin) Unreserve(_ context.Context, _ *framework.CycleState, _ *v1.Pod, _ string) { } // NewFakeReservePlugin initializes a fakeReservePlugin and returns it. func NewFakeReservePlugin(status *framework.Status) frameworkruntime.PluginFactory { return func(_ runtime.Object, _ framework.FrameworkHandle) (framework.Plugin, error) { return &FakeReservePlugin{ Status: status, }, nil } } // FakePreBindPlugin is a test prebind plugin. type FakePreBindPlugin struct { Status *framework.Status } // Name returns name of the plugin. func (pl *FakePreBindPlugin) Name() string { return "FakePreBind" } // PreBind invoked at the PreBind extension point. func (pl *FakePreBindPlugin) PreBind(_ context.Context, _ *framework.CycleState, _ *v1.Pod, _ string) *framework.Status { return pl.Status } // NewFakePreBindPlugin initializes a fakePreBindPlugin and returns it. func NewFakePreBindPlugin(status *framework.Status) frameworkruntime.PluginFactory { return func(_ runtime.Object, _ framework.FrameworkHandle) (framework.Plugin, error) { return &FakePreBindPlugin{ Status: status, }, nil } } // FakePermitPlugin is a test permit plugin. type FakePermitPlugin struct { Status  *framework.Status Timeout time.Duration } // Name returns name of the plugin. func (pl *FakePermitPlugin) Name() string { return "FakePermit" } // Permit invoked at the Permit extension point. func (pl *FakePermitPlugin) Permit(_ context.Context, _ *framework.CycleState, _ *v1.Pod, _ string) (*framework.Status, time.Duration) { return pl.Status, pl.Timeout } // NewFakePermitPlugin initializes a fakePermitPlugin and returns it. func NewFakePermitPlugin(status *framework.Status, timeout time.Duration) frameworkruntime.PluginFactory { return func(_ runtime.Object, _ framework.FrameworkHandle) (framework.Plugin, error) { return &FakePermitPlugin{ Status:  status, Timeout: timeout, }, nil } } </ins>

This is a bug found in The scheduler fails to schedule the pod on the first try due to PV is out of sync. This may happen because the PV/PVC may be modified before the scheduler binder assumes them. However, the scheduler skips the pod update event and never try to scheduler the pod again: This is because when the reserve plugins fails, we didn't try to forget the pod. After the is called, the pod is in the state.
/sig scheduling /sig storage /assign /priority critical-urgent
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig scheduling /sig storage /assign /priority critical-urgent Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "strings" "sync" "testing" <ins> "time" </ins> "github.com/stretchr/testify/assert"

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I am running etcd-operator in kubernetes, and noticed that when I deleted the etcdcluster (Custom resource definition) the pods & services did not get cleaned up. I looked on another kube environment I had running, and on that env everything worked as expected. I checked logs, and could see on the env that was working I saw the following messages in Controller manager logs: I didn't see these in the env that wasn't working, so I started looking at the garbage collector on there. I noticed that there hadn't been any log message from garbage collecor since 31st Jan, when this was logged: Around this time we experienced some environmental network issues, which caused lots of errors around that time, such as: It looks like these issues may have caused the garbage collector to get into some sort of hung state? When I restarted controller manager everything started working as expected. What you expected to happen: Garbage collector should have kept running, and working as expected. How to reproduce it (as minimally and precisely as possible): Not sure on the exact sequence that caused it, but suspect it was related to a time period when errors were occurring. Anything else we need to know?: We have set, and had previously hit this garbage collector issue - However, we are running with Kube 1.8.6 with the fix for that issue cherry-picked (commit ) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: IBM Cloud OS (e.g. from /etc/os-release): Ubuntu "16.04.3 LTS (Xenial Xerus)" Kernel (e.g. ): 0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: I will attach logs from Controller manager.
/sig api-machinery
/assign /assign
cc
I saw garbage collector stopped working in 1.9.2 too. Most controllers (such as node controllers or deployment controllers) stopped reporting logs in controller- when this happened. Manually restarting controller-manager fixed the problem. I think somehow some controllers got stuck (could it be because of some bugs in watch cache?) and it's not specific to garbage collector.

When the issue happened, still reported healthy for controller-manager.
tracks this
This is happening in production for us too.
did this happen to you to? Or is it that only the garbage collector disappeared from the logs?
For me the other controllers still have logs; it was just the GC that crashed.
does your cluster also use custom resources backed by Aggregated API servers?
Yes, although this happened in one cluster where we don't have any resources yet (the type exists). I'm not sure what that means
I still see messages from and and after the garbage collector stops working - so looks like some controllers are still working. Is there anything specific you were looking for?
I think the cause of this garbage collector hanging here is this: function periodically calls to the comment above GetDeletableResources, , an error in discovery causes the discovery client to no longer discover resources in the cluster, but instead of failing and returning an error, it simply logs the error as and returns an empty list of resources Sync function, upon recieving an empty resource list from discovery, detects that the resources have changed, and calls , which calls with as the argument as shown in the log as , which sets the list of monitors to an empty list because it thinks there are no resources to monitor. the Sync function calls controller.WaitForCacheSync, which calls , which will continually retry the function until it returns true, but it will always return false because is 0. So I think this was caused by which enabled number 2.