Question: <del> #!/bin/bash KUBE_HOOKS_DIR="$(dirname "$(test -L "$0" && echo "$(dirname $0)/$(readlink "$0")" || echo "$0")")" files_need_gofmt=() files_need_boilerplate=() files_need_description=() files=($(git diff --cached --name-only --diff-filter ACM | grep ".go" | grep -v -e "third_party" -e "Godeps")) for file in "${files[@]}"; do # Check for files that fail gofmt. diff="$(git show ":${file}" | gofmt -s -d)" if [[ -n "$diff" ]]; then files_need_gofmt+=("${file}") fi done boiler="${KUBE_HOOKS_DIR}/boilerplate.py" # Check for go files without the required boilerplate. if [[ ${#files[@]} -gt 0 ]]; then files_need_boilerplate+=($("${boiler}" "go" "${files[@]}")) fi # Check for sh files without the required boilerplate. files=($(git diff --cached --name-only --diff-filter ACM | grep ".sh" | grep -v -e "third_party" -e "Godeps")) if [[ ${#files[@]} -gt 0 ]]; then files_need_boilerplate+=($("${boiler}" "sh" "${files[@]}")) fi # Check for py files without the required boilerplate. files=($(git diff --cached --name-only --diff-filter ACM | grep ".py" | grep -v -e "third_party" -e "Godeps")) if [[ ${#files} -gt 0 ]]; then files_need_boilerplate+=($("${boiler}" "py" "${files[@]}")) fi # Check API schema definitions for field descriptions for file in $(git diff --cached --name-only --diff-filter ACM | egrep "pkg/api/v.[^/]*/types.go" | grep -v "third_party"); do # Check for files with fields without description tags descriptionless=$("${KUBE_HOOKS_DIR}/description.sh" "${file}") if [[ "$descriptionless" -eq "0" ]]; then files_need_description+=("${file}") fi done if [[ "${#files_need_gofmt[@]}" -ne 0 ]]; then ( echo echo "# *** ERROR: *** Some files have not been gofmt'd.  To fix these" echo "# errors, run gofmt -s -w <file>, or cut and paste the following:" echo "#   gofmt -s -w ${files_need_gofmt[@]}" echo "#" echo "# Your commit will be aborted unless you override this warning. To" echo "# commit in spite of these format errors, delete the following line:" echo "#   COMMIT_BLOCKED_ON_GOFMT" ) >> $1 fi if [[ "${#files_need_boilerplate[@]}" -ne 0 ]]; then ( echo echo "# *** ERROR: *** Some files are missing the required boilerplate" echo "# header from hooks/boilerplate.txt:" for file in "${files_need_boilerplate[@]}"; do echo "#   ${file}" done echo "#" echo "# Your commit will be aborted unless you fix these." echo "#   COMMIT_BLOCKED_ON_BOILERPLATE" echo ) >> $1 fi if [[ "${#files_need_description[@]}" -ne 0 ]]; then ( echo echo "# *** ERROR: *** Some API files are missing the required field descriptions" echo "# Add description tags to all non-inline fields in the following files:" for file in "${files_need_description[@]}"; do echo "#   ${file}" done echo "#" echo "# Your commit will be aborted unless you fix these." echo "#   COMMIT_BLOCKED_ON_DESCRIPTION" echo ) >> $1 fi if ! hack/verify-gendocs.sh > /dev/null; then ( echo echo "# *** ERROR: *** docs are out of sync between cli and markdown" echo "# run hack/run-gendocs.sh > docs/kubectl.md to regenerate" echo echo "#" echo "# Your commit will be aborted unless you regenerate docs." echo "    COMMIT_BLOCKED_ON_GENDOCS" echo ) >> $1 fi </del>

Multiple, for example: Since See above The test tolerations are now to strict: No response /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think it's from if you check the cri-o versions the succeessful one was and the failing was
Working on a fix in
cc
/reopen need to bump in kube as well to fully get the fix
Reopened this issue. <detailsIn response to : /reopen </details


Question: spec: containers: - name: kubedns <del> image: gcr.io/google_containers/kubedns-amd64:1.5 </del> <ins> image: gcr.io/google_containers/kubedns-amd64:1.7 </ins> resources: # TODO: Set memory limits when we've profiled the container for large # clusters, then set request = limit to keep this container in

Now that merged, we need to build a new kube-dns release.
I just put in a PR yesterday to fix the missing changelog entry for kube-dns:1.6


Question: scheduler := makeScheduler(predicates, nodes) pod := &v1.Pod{ObjectMeta: metav1.ObjectMeta{Name: "1", UID: types.UID("1")}} <del> _, predicateMap, err := scheduler.findNodesThatFit(nil, pod, nodes) </del> <ins> _, predicateMap, err := scheduler.findNodesThatFit(nil, pod, schedulertesting.FakeNodeLister(nodes)) </ins> if err != nil { t.Errorf("unexpected error: %v", err)

In : It seems that nodes are only used in the case there are no predicates. This call can be moved inside . Moving it there saves us from creating the node slice unless it is necessary. Checking if there are no nodes (line 202 below) can be done using . /kind cleanup /sig scheduling
I can work on this


Question: lun, err := c.GetNextDiskLun(nodeName) if err != nil { klog.Warningf("no LUN available for instance %q (%v)", nodeName, err) <del> return fmt.Errorf("all LUNs are used, cannot attach volume (%s, %s) to instance %q (%v)", diskName, diskURI, instanceid, err) </del> <ins> return -1, fmt.Errorf("all LUNs are used, cannot attach volume (%s, %s) to instance %q (%v)", diskName, diskURI, instanceid, err) </ins> } klog.V(2).Infof("Trying to attach volume %q lun %d to node %q.", diskURI, lun, nodeName) <del> return vmset.AttachDisk(isManagedDisk, diskName, diskURI, nodeName, lun, cachingMode) </del> <ins> return lun, vmset.AttachDisk(isManagedDisk, diskName, diskURI, nodeName, lun, cachingMode) </ins> } // DetachDisk detaches a disk from host. The vhd can be identified by diskName or diskURI.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig azure /assign


Question: {kubeNodePortLocalSetSCTP, string(KubeNodePortChain), "RETURN", "dst", "sctp"}, } <del> var ipvsModules = []string{ "ip_vs", "ip_vs_rr", "ip_vs_wrr", "ip_vs_sh", "nf_conntrack_ipv4", "nf_conntrack", } </del> // In IPVS proxy mode, the following flags need to be set const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--kube-proxy cannot use ipvs mode on Linux 4.19 due to a check for the module, which was changed to (). uses to check for : It should instead accept on Linux 4.19 or above. What happened: kube-proxy fell back from ipvs mode to iptables mode. What you expected to happen: kube-proxy using ipvs. How to reproduce it (as minimally and precisely as possible): Launch a cluster on an image using Linux 4.19 or above with kube-proxy using ipvs mode (see ). Environment: Kubernetes version (use ): All Kernel (e.g. ): 4.19 <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig network
/area ipvs /assign /cc
Okay, so should check {nfconntrackipv4 | nf_conntrack} and expect load at least one module?
I think so
Please just go ahead :)


Question: if err != nil { return nil, err } <del> s.versioner.UpdateObject(state.obj, uint64(rv)) </del> <ins> if err := s.versioner.UpdateObject(state.obj, uint64(rv)); err != nil { klog.Errorf("failed to update object version: %v", err) } </ins> return state, nil }

Remove panic in , please return error instead following .
/sig api-machinery /good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /sig api-machinery /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Please also add some log if returns error
/remove-kind bug /kind cleanup


Question: path: ../_output/local/bin/logcheck.so description: structured logging checker original-url: k8s.io/logtools/logcheck <ins> forbidigo: analyze-types: true forbid: - p: ^managedfields.ExtractInto$ pkg: ^k8s.io/apimachinery/pkg/util/managedfields$ msg: should not be used because managedFields was removed - p: .Extract pkg: ^k8s.io/client-go/applyconfigurations/ msg: should not be used because managedFields was removed </ins> gocritic: enabled-checks:             # not limited in golangci-strict.yaml - equalFold               # not limited in golangci-strict.yaml

Add a code static check for to report using server-side-apply extract With optimization of and , the will be removed in and , and maybe more of other components in the future, using Extract is problematic. It's safer to add this lint check.
/cc
/sig api-machinery
/sig scheduling /sig apps
/assign /triage accepted


Question: // GetEndpointNodes returns a map of nodenames:external-ip on which the // endpoints of the Service are running. func (j *TestJig) GetEndpointNodes() (map[string][]string, error) { <del> nodes, err := e2enode.GetBoundedReadySchedulableNodes(j.Client, MaxNodesForEndpointsTests) </del> <ins> nodes, err := j.ListNodesWithEndpoint() </ins> if err != nil { return nil, err } <del> epNodes, err := j.GetEndpointNodeNames() </del> <ins> nodeMap := map[string][]string{} for _, node := range nodes { nodeMap[node.Name] = e2enode.GetAddresses(&node, v1.NodeExternalIP) } return nodeMap, nil } // ListNodesWithEndpoint returns a list of nodes on which the // endpoints of the given Service are running. func (j *TestJig) ListNodesWithEndpoint() ([]v1.Node, error) { nodeNames, err := j.GetEndpointNodeNames() </ins> if err != nil { return nil, err } <del> nodeMap := map[string][]string{} for _, n := range nodes.Items { if epNodes.Has(n.Name) { nodeMap[n.Name] = e2enode.GetAddresses(&n, v1.NodeExternalIP) </del> <ins> ctx := context.TODO() allNodes, err := j.Client.CoreV1().Nodes().List(ctx, metav1.ListOptions{}) if err != nil { return nil, err } epNodes := make([]v1.Node, 0, nodeNames.Len()) for _, node := range allNodes.Items { if nodeNames.Has(node.Name) { epNodes = append(epNodes, node) </ins> } } <del> return nodeMap, nil </del> <ins> return epNodes, nil </ins> } // GetEndpointNodeNames returns a string set of node names on which the

Example failure: Stacktrace: The problem seem to be related to the fact that we're running the test on private cluster (nodes doesn't have external IP address).
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Given that there are more issues with the NetLB tests than just this particular one, I think we should revert my PR that re-enabled them. ! Wojtek just sent a PR to do that -
But let's leave this issue open - we want to fix and re-enable it again.
The revert merged, but we didn't managed to get it in before the yesterday's run started. Today's run should be green.
Yeah - but that doesn't mean we should close it later - we should fix and reenable this test. The issue should be opened until that test is reenabled.
That's clear, I didn't say anywhere that we should close it
Ok, the tests have been disabled and they don't fail the correctness tests anymore. When it comes to fixing this particular test. I took a cursory glance and I think it is fixable. We should change the code to not expect public IPs on the nodes. And then when poking over http we should do it not from the test but proxy via some container. There is already a code written for that, see /help wanted
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/remove-triage unresolved
I would like to take this /assign
- thanks!

contains brief description on how to fix that
so are we going to access all the nodes via a test container?
Not sure if I fully understand your question, so if you meant something different let me know: to make the test work in large clusters (5k nodes), we can't access all - we may want to do tricks we do in other tests like access min(cluster size, some constant (e.g. 10 or 100) if you meant "via a test container" - yes, I think it's fine to just migrate the test to always use it
Reopening - since we need to reenable this test.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Does this issue need an owner? I got a little lost wading through all of the issues/PRs attached to this issue. If I wanted to take up the reins, how do I trigger the 5k node test on a PR or is it automatically triggered?
The test is disabled. I think the crucial part is done now - which is: I can't remember what else left to be done. Unfortunately you can't do that from oss. We only have post-submits exposes as these are too expensive. We can try running it internally to check if it now works...
We reenabled this test and it seem to work. So closing this one. Thanks for all the help on it!


Question: <del> #!/bin/bash # Copyright 2015 The Kubernetes Authors All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. set -o errexit set -o nounset set -o pipefail set -o xtrace export PATH=${GOPATH}/bin:${PWD}/third_party/etcd:/usr/local/go/bin:${PATH} ./hack/install-etcd.sh go get -u github.com/jstemmer/go-junit-report go get golang.org/x/tools/cmd/cover go get github.com/mattn/goveralls go get github.com/tools/godep go get github.com/jstemmer/go-junit-report # Enable the Go race detector. export KUBE_RACE=-race # Disable coverage report export KUBE_COVER="n" # Produce a JUnit-style XML test report for Jenkins. export KUBE_JUNIT_REPORT_DIR=${WORKSPACE}/artifacts # Save the verbose stdout as well. export KUBE_KEEP_VERBOSE_TEST_OUTPUT=y export KUBE_GOVERALLS_BIN="${GOPATH}/bin/goveralls" export KUBE_TIMEOUT='-timeout 300s' export KUBE_COVERPROCS=8 export KUBE_INTEGRATION_TEST_MAX_CONCURRENCY=4 export LOG_LEVEL=4 ./hack/verify-gofmt.sh ./hack/verify-boilerplate.sh ./hack/verify-description.sh ./hack/verify-flags-underscore.py ./hack/verify-generated-conversions.sh ./hack/verify-generated-deep-copies.sh ./hack/verify-generated-docs.sh ./hack/verify-generated-swagger-docs.sh ./hack/verify-swagger-spec.sh ./hack/verify-linkcheck.sh ./hack/test-go.sh -- -p=2 ./hack/test-cmd.sh ./hack/test-integration.sh ./hack/test-update-storage-objects.sh </del>

In the pull request Jenkins, we're running up to 8 instances of verification checks, build, unit tests, integration tests, and e2e tests. Unfortunately, it seems like several of these components were not designed to run simultaneous to other tests on the same machine. In particular, several unit tests seem to write to static file names in /tmp (or equivalent), which is a fairly easy fix. Some of the verification checks and the integration test start up etcd, however, and the etcd helper checks that only a single etcd binary is running on the host (and uses a static port for it). For example: We might be able to fix the etcd issue separately, or we can just figure out how to containerize all of these tests so they can't possibly interfere with each other. Note that we're ignoring the exit status of the verification checks and unit/integration tests right now, so these etcd failures are effectively being ignored (though it is causing us to miss test coverage).
What percentage of tests are failing due to being non-hermetic like this?
I'm not sure about unit tests (I've anecdotally seen just a few fail), but the verification checks are failing fairly regularly, and the integration test is also similarly frequently not running due to this.
To be clear, I'm looking for real numbers, to decide whether it's feasible to temporarily disable a small number of tests until we can hermetify them. Otherwise we need to turn off all unit, integration and verification tests in Jenkins to avoid irritating invalid regression signals. We need to do one or the other this week.
I'll try to dig through the logs, figure out which ones have failed recently, disable them in Jenkins, and see whether the flakes abate.
I just turned off all unit, integration and verification checks in Jenkins, as a temporary measure to unstall the merge-bot.
As a point of information, I just found approximately 20 tests in which use a static tmp path with semi-random filenames which are highly likely to collide. We could keep searching for unit tests that are non-hermetic, but it'd probably be a better investment of time to fix the test infrastructure so test authors don't need to care (and so we don't need to go look at and fix ~2000 test cases).

Another few metrics, using the string "/tmp" as a likely anti-pattern: We'd need to update each of these 23 files to get them to use or if they aren't already.
Created to track cleaning up tempdir usage in unit tests.
cc


Question: seenPlugins: make(map[string]string, 0), } <del> path := "PATH" if runtime.GOOS == "windows" { path = "path" } o.PluginPaths = filepath.SplitList(os.Getenv(path)) </del> <ins> o.PluginPaths = filepath.SplitList(os.Getenv("PATH")) </ins> return nil }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, the kubectl plugins system - when running under Windows - will determine that a plugin is executable by looking at the plugin's extension, comparing it to '.exe'. I think we should allow plugins w/ a few more known extensions (.bat, .com, .cmd; for example). Why is this needed: This would give more of a similar experience to plugins under *nix, where shell scripts are often used.
/sig cli
Please see the PR below:


Question: ctx, cancel := getContextWithCancel() defer cancel() <del> glog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk", nodeResourceGroup, nodeName) resp, err := ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm) if ss.CloudProviderBackoff && shouldRetryHTTPRequest(resp, err) { glog.V(2).Infof("azureDisk - update(%s) backing off: vm(%s)", nodeResourceGroup, nodeName) retryErr := ss.UpdateVmssVMWithRetry(ctx, nodeResourceGroup, ssName, instanceID, vm) if retryErr != nil { err = retryErr glog.V(2).Infof("azureDisk - update(%s) abort backoff: vm(%s)", nodeResourceGroup, nodeName) } } if err != nil { </del> <ins> glog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s)", nodeResourceGroup, nodeName, diskName) if _, err := ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, vm); err != nil { </ins> detail := err.Error() if strings.Contains(detail, errLeaseFailed) || strings.Contains(detail, errDiskBlobNotFound) { // if lease cannot be acquired or disk not found, immediately detach the disk and return the original error <del> glog.Infof("azureDisk - err %s, try detach", detail) </del> <ins> glog.Infof("azureDisk - err %s, try detach disk(%s)", detail, diskName) </ins> ss.DetachDiskByName(diskName, diskURI, nodeName) } } else { <del> glog.V(4).Info("azureDisk - azure attach succeeded") </del> <ins> glog.V(2).Infof("azureDisk - attach disk(%s) succeeded", diskName) </ins> // Invalidate the cache right after updating key := buildVmssCacheKey(nodeResourceGroup, ss.makeVmssVMName(ssName, instanceID)) ss.vmssVMCache.Delete(key)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: retry azure disk attach operation produce too many API calls Not the default value is 6 which means if AttachDisk operation failed, it will retry 6 more times, when disk quota is full on a VM, it produces too many API calls. Retry attach disk behavior is not necessary since it AttachVolume operation fails, k8s attachdetach-controller will retry once a minute until it succeeds. default setting in What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.10, v1.11, v1.12 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig azure /assign


Question: "k8s.io/klog" apierrors "k8s.io/apimachinery/pkg/api/errors" <ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" </ins> "k8s.io/apimachinery/pkg/labels" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait"

k8s version: v1.9.3 When I create customresourcedefinition, There is error like Operation cannot be fulfilled on "": the object has been modified; please apply your changes to the latest version and try again . apiVersion: kind: CustomResourceDefinition metadata: name: annotations: "": crd-install labels: app: istio-pilot spec: group: names: kind: VirtualService listKind: VirtualServiceList plural: virtualservices singular: virtualservice categories: istio-io networking-istio-io scope: Namespaced version: v1alpha3 So what is the reason of it?
a few questions: where are you seeing the error, in the apiserver log or in the output of the kubectl command? what kubectl command are you using to create the object? are you running HA apiservers?
The log is shown by apiserver. I run k8s in HA apiservers. The command I used is "kubectl apply -f "+file.
API servers attempt to update CRD status to indicate whether the names were available. In an HA apiserver setup, the API servers race to do so, and all but the first lose the race with a resource conflict error. That is not unexpected. Opened to avoid logging normal API responses in that scenario.
Thanks ! But I want to know whether the CRD has been set successfully in HA apiservers ? I use the command " kubectl get crd -o yaml" and get the following results. ! I see its status is true. But when i create a virtualservices, it returns error. Error from server (Timeout): error when creating "cms-api-": Timeout: request did not complete within allowed duration ! So i want to know how to solve this error.
/assign
Excuse me. Do you have some greate ideas to solve this issue??
There is a PR for this sent by
I see your request is timed-out not failed. In HA, I expect the request to fail in the first seconds of the CRD creation, but after that it should go through. I need more logs during this timeout or a consistent way to reproduce it.


Question: fit bool, reasons []algorithm.PredicateFailureReason, equivalenceHash uint64, <ins> needLock bool, </ins> ) { <del> ec.Lock() defer ec.Unlock() </del> <ins> if needLock { ec.Lock() defer ec.Unlock() } </ins> if _, exist := ec.algorithmCache[nodeName]; !exist { ec.algorithmCache[nodeName] = newAlgorithmCache() }

Fix this TODO (raised by : Part of: One thoughts to fix this is, we can redesign the equivalence cache data structure to: bool instead of deleting the entry. certain version tag in the entry, update the version per WRITE op (Update and Invalidate). the version after eCache predicate, and check it before update eCache.
Another possible approach is: eCache before running a predicate. the predicate. the eCache for the predicate. eCache. Basically, in the existing code all predicates are run first and their results are collected and then at the end of the function the eCache is updated. Instead of that, we can move the eCache update for each predicate inside the for loop and next to each predicate execution and synchronize before start of the execution of each predicate. The downside to my suggested approach is that performance may suffer due to frequent locking and unlocking.
Prefer to bsalamat@ 's option which is simpler :). Comparing to performance improvement by eCache, the lock is acceptable to me.
yup same here, going with the locking is easier and is acceptable. - also using a flag for invalidation might just move the problem from eCache to the flag ?
Oh I missed this notification somehow, busy on cleaning scheduler integration test. Yep so if locking is acceptable, it should be cleanest way to fix this, optimistic control would be a more complicated story.
- are you handling this one ?
Already been taken by volunteer days ago :)


Question: // failed by this plugin schedulable. func (pl *NodeAffinity) EventsToRegister() []framework.ClusterEvent { return []framework.ClusterEvent{ <del> {Resource: framework.Node, ActionType: framework.Add | framework.UpdateNodeLabel}, </del> <ins> {Resource: framework.Node, ActionType: framework.Add | framework.Update}, </ins> } }

We have an e2e test that involves cluster autoscaler; the test creates two pods back to back which forces CA to scale up the cluster to create two nodes also back to back one for each pod. The two nodes are not created at the same time, there is some delay between them. The problem is that the second pod never gets scheduled. We see in the logs a single scheduling attempt for the second pod that attempted to schedule it on the first node and failed because it didn't fit the node's remaining cpu cores. We don't see any attempts after that for the pod and the test times out waiting for that pod to schedule. What changed in the scheduler: Pre 1.22, on most update/add/delete events to pods/nodes/pvc/storageClass resources, we used to move all unschedulable pods back to the queue. Also every we did force move all unschedulable pods back to the queue indiscriminately. This behavior is not efficient, wasted scheduling cycles and since we re-queue based on last scheduling attempt timestamp, it missed up badly with the best effort FIFO behavior we intend to have for pod scheduling.Since 1.22 a couple of things changed such that unschedulable pods are not placed back into the queue on those events indiscriminately; on each such event, we iterate over unschedulable pods and we place them back into the queue if they pass two checks: the first one happens only on node update/add events, it is carried by the function ; it basically runs the kubelet admission logic (node resource, affinity, taints/tolerations and host port checks) the second check runs on all events (including the node ones); this one is a little more complicated and fine grained: for each unschedulable pod we remember which plugins last failed to fit the pod; at the same time plugins register which events could make an unschedulable pod schedulable (e.g. nodeResources filter plugin registers pod delete and node add/update events). A pod is put back into the queue only if the event matches any of the ones registered by the plugins that last failed the pod.

In 1.24 we increased the 60s to (our north star to completely remove this) What went wrong: My theory is that we incorrectly determined that the pod was not eligible to be put back into the queue on events related to the new node; and that the 5min timeout was too long to force moving it back to the queue (I am guessing the whole test timed out), while the 60s was basically masking the problem for 1.22 and 1.23. So what checks are incorrectly failing to move the pod? Here is what happened: Before adding the new node, the pod failed the nodeResource plugin, which only registers for New node was , scheduler receives an add event. The pod was not moved back to the queue because it failed the check on NotReady taint. When the new node became ready, the scheduler receives a node update event, but since it is a taint update event, it didn't match the events that the nodeResources plugin registered for. How to fix it: Since we have the ; pods that previously failed because of NodeResources, NodePorts, TaintsAndTolerations and NodeAffinity (the predicates we already ran in ) should always be placed back into the queue; basically those plugins should register for all node update events. /sig scheduling /cc The second pod to schedule. See above. No response 24 GKE <details</details<details</details<details</details<details</details
/triage accepted


Question: } }) <del> ginkgo.It("should ensure that requests can't be drowned out", func() { </del> <ins> // This test creates two flow schemas and a corresponding priority level for // each flow schema. One flow schema has a higher match precedence. With two // clients making requests at different rates, we test to make sure that the // higher QPS client cannot drown out the other one despite having higher // priority. ginkgo.It("should ensure that requests can't be drowned out (priority)", func() { </ins> flowSchemaNamePrefix := "e2e-testing-flowschema" priorityLevelNamePrefix := "e2e-testing-prioritylevel" loadDuration := 10 * time.Second

Should we create tests around the creation, modification and deletion of the objects ? For example, adding an invalid FlowSchema with a misconfigured PriorityLevelConfiguration and verifying that the Status appropriately reflects its invalidity. I’m assuming the controller tests will suffice and we don’t need explicit e2e tests for this. What about CRUD tests wherein we’re validating the debug endpoints are showing the right values ? The first class of tests are the basic ones to verify that the requests are being categorized into the right FlowSchema and Priority level. We’ll set up a new PriorityLevelConfiguration and FlowSchema objects and observe that requests are correctly being classified onto them. Verification that requests are correctly being classified will be tricky. For now, we can scrape the metrics endpoint for ‘’ metric to verify that the expected number of requests for a test FlowSchema is correct. Another option is to utilize the ‘dump_requests’ debug endpoint to verify the correct request classification. If during the course of writing the tests we need additional information, we can extend the debug endpoint. Scraping the metrics endpoints to validate the proper request categorization could lead to a fickle e2e test. So querying the ‘debug’ endpoint seems like the right way to go. This test will also serve as a proxy for an end-to-end test that ensures that requests matching a new FlowSchema are successful. Does this have the propensity to be a flaky test(s) ? This class of tests will focus on validating the behavior this feature promises, viz., requests of low priority won’t be drowned out by requests of higher priority and a single bad actor won’t detrimentally affect the requests from others. The following tests will ensure that high velocity flows don’t affect the low velocity flows. For this test, we’ll have two entities submitting requests at the same time. These requests will map to two different FlowSchemas at different priority levels. The higher priority flow will have a lot of requests going through it, far above its concurrency share. At the same time the low priority flow will be submitting requests as well at a rate that well within the concurrency limits of that Flow. The test will ensure that the low velocity (low priority) flow succeeds. This test will be the same as the above one except that the two flows will map to the same FlowSchema but are separated via their Flow Distinguisher.

One of them will be a high velocity flow which the other is a low velocity one. The test will ensure that the low velocity requests still get through and succeed. Does this have the propensity to be a flaky tests ? If the test fails, will the runner of this test be able to grok where the problem could lie and debug ? Is this an actionable failure ?
cc/
We don't need to test that invalid items are rejected, unit tests are better for that. We should try to exercise the basic features (exempt, queue...) We should not rely on the debug endpoint, since we want this to be a conformance test eventually Metrics scraping is also a little questionable. I think we should exercise fair queueing (make two new PLs at different shares, saturate both, verify the responses follow the share ratio). I think we should exercise the shuffle sharding (make a new PL, saturate it with one/few elephants and many mice, verify the mice get service. (those latter two seem to match your last two.) This would be a pretty high level test that tests things end users will care about, but I think it should still be quite hard to pass without actually running our code.
/sig api-machinery /area test /kind feature
cc
to verify the requests can be correctly classified, how about we make use of the ? we can do a test like: make two PLs, and two clients w/ different identities then make the clients are hitting different FS/PL.
what about using a test aggregated apiserver to serve different PLs, and with different the response latency
No strong opinion on this one. I am not sure if that was a response to the ideas in the "Basic CRUD" section --- there the idea is not about testing validation but rather testing the status setting about referential integrity. The debug endpoints are about what is happening at the moment of the GET. Controlling that tightly enough to be worth a test would be tricky. The only plausible approach in an E2E test would be to register an authorization webhook that synchronizes with the test function. I like the basic idea. For the reasons above, I would not recommend looking at the debug endpoints. I do not understand what is wrong with scraping the metrics. Are they not part of conformance? This feature does add response headers.

Are those part of conformance? I like the basic idea. First, remember that this feature does not control velocity; it controls the product of velocity and execution duration. To get a high signal-to-noise ratio, we would want deterministic and large execution durations. Looking in , you can see that authorization is included in execution. So we could add a controlled large amount of duration in there. I am a little surprised at the question of "If the test fails, will the runner of this test be able to grok where the problem could lie and debug"; why would that be more of a question here than for any other E2E test? Presumably the test failure message will say something about the expected and observed behavior. I think the drown-out scenario involves testing queuing. I think exempt was not covered above, and would be a worthwhile addition. This mixes up two things. Fair queuing is something that happens within one priority level. This sounds like it may be expensive for an E2E test. I think we can make a much more efficient unit test of QueueSet that exercises shuffle sharding. Because the QueueSet implementation can be given a FakeEventClock, we can run the test much faster than real time. To get a high signal-to-noise ratio we want controlled, high latencies. Making an aggregated apiserver do that is possible. Perhaps the simplest approach would be to put a fixed sleep in the registry strategies for whatever custom resources the aggregated apiserver is serving. But this seems like a lot more work than necessary. I think an authorization webhook is a simpler approach.
could you elaborate a bit more about the authorization webhook approach? it is only called in the authorization path, if i understand it correctly.
Authorization, and admission control, happen after the priority and fairness filter.
is this really possible? I am likely missing something, but my understanding from the is that each level is limited to the concurrency shares assigned to it.
: the text should be revised to more clearly express the intent. It is that the driving client has many more requests outstanding at once than the corresponding priority level will allow to be served concurrently.
FYI: I'm working on the second test.


Question: return nil, rerr.Error() } <del> for _, scaleSet := range allScaleSets { </del> <ins> for i := range allScaleSets { scaleSet := allScaleSets[i] </ins> if scaleSet.Name == nil || *scaleSet.Name == "" { klog.Warning("failed to get the name of VMSS") continue

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Controller manager makes many invalid VMSS update calls. What you expected to happen: Controller manager makes correct VMSS update calls. How to reproduce it (as minimally and precisely as possible): Use AKS-Engine to deploy a cluster on Azure. Anything else we need to know?: Environment: Kubernetes version (use ): 16.6 Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/area provider/azure /sig cloud-provider


Question: func New(config *azclients.ClientConfig) *Client { baseURI := config.ResourceManagerEndpoint authorizer := config.Authorizer <del> armClient := armclient.New(authorizer, baseURI, config.UserAgent, APIVersion, config.Location, config.Backoff) </del> <ins> apiVersion := APIVersion if strings.EqualFold(config.CloudName, AzureStackCloudName) { apiVersion = AzureStackCloudAPIVersion } armClient := armclient.New(authorizer, baseURI, config.UserAgent, apiVersion, config.Location, config.Backoff) </ins> rateLimiterReader, rateLimiterWriter := azclients.NewRateLimiter(config.RateLimitConfig) klog.V(2).Infof("Azure StorageAccountClient (read ops) using rate limit config: QPS=%g, bucket=%d",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : PR() already proves that old api-version on Azure stack works on global Azure too, we could support azure stack in azure cloud provider lib, and then make sure following components works in Azure stack: CSI drivers: azure disk, azure file CCM Since it's mainly api-version change, we could also add e2e tests against global Azure too. Why is this needed: /assign /priority important-soon /sig cloud-provider /area provider/azure
there is blocking issue here(also need azure stack supports latest data model struct): just making the change in interfaces and not go-sdk but facing this error, since we don't have a switch inside go-sdk, we only do switch in azure cloud provider lib,
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package preflight import ( "testing" "time" utilwait "k8s.io/apimachinery/pkg/util/wait" ) func TestParseServerURIGood(t *testing.T) { host, err := parseServerURI("https://127.0.0.1:2379") if err != nil { t.Fatalf("unexpected error: %v", err) } reference := "127.0.0.1:2379" if host != reference { t.Fatal("server uri was not parsed correctly") } } func TestParseServerURIBad(t *testing.T) { _, err := parseServerURI("-invalid uri$@#%") if err == nil { t.Fatal("expected bad uri to raise parse error") } } func TestEtcdConnection(t *testing.T) { etcd := new(EtcdConnection) result := etcd.serverReachable("-not a real network address-") if result { t.Fatal("checkConnection should not have succeeded") } } func TestCheckEtcdServersEmpty(t *testing.T) { etcd := new(EtcdConnection) result, err := etcd.CheckEtcdServers() if err != nil { t.Fatalf("unexpected error: %v", err) } if result { t.Fatal("CheckEtcdServers should not have succeeded") } } func TestCheckEtcdServersUri(t *testing.T) { etcd := new(EtcdConnection) etcd.ServerList = []string{"-invalid uri$@#%"} result, err := etcd.CheckEtcdServers() if err == nil { t.Fatalf("expected bad uri to raise parse error") } if result { t.Fatal("CheckEtcdServers should not have succeeded") } } func TestCheckEtcdServers(t *testing.T) { etcd := new(EtcdConnection) etcd.ServerList = []string{""} result, err := etcd.CheckEtcdServers() if err != nil { t.Fatalf("unexpected error: %v", err) } if result { t.Fatal("CheckEtcdServers should not have succeeded") } } func TestPollCheckServer(t *testing.T) { err := utilwait.PollImmediate(1*time.Microsecond, 2*time.Microsecond, EtcdConnection{ServerList: []string{""}}.CheckEtcdServers) if err == nil { t.Fatal("expected check to time out") } } </ins>

See: This code should make several attempts before ing. Alternatively, we could consider not listening for connections until this code has succeeded.
Feel free to correct me if I'm wrong on any of this since I've never looked at kubernetes code base before. The code for the and the both appear to make multiple attempts before returning an error which leads to a Fatal.
, I think we had waited for to start at . Do we need to wait more or make it configurable?
I would like to work on this if nobody else is.
No need to ask for permission! Just say here that you're working on it, that should be sufficient to prevent duplicate work. :)
Can I have a go at this?
I have a nearly completed solution for this already. Just working on testing/test cases. WIll hopefully sumbit a PR shortly.
Ok.
I wondered why we care about this. After all, kubelet restarts failing pods. For posterity, after talking with the reason is that the apiserver may fail so fast that the kubelet begins to back off on restarts of a failing pod. That backoff may be too aggressive. (Maybe it causes spurious alerts, or maybe it allows too much downtime.) So, let's fix the apiserver. That's fine, but clearly we're not the only user that might encounter this. You may be running a server you don't want to (or can't) modify, and you might wish you could tweak the kubelet's backoff heuristics for your pod. Alternately, maybe the kubelet could choose not to enter the backoff if apiserver exited with a different code. anything like this already in place?
That's a wonderful explanation that hadn't occurred to me. If there isn't a feature request for this already I'll work on something and reference this issue.
AFAIK, the backoff mechanism is there to protect kubelet from being overwhelmed by failed user containers. Trusting the containers with returning a specific exit code sounds like a big "no" to me. Making this configurable per pod might be ok, but it's still not good in general to allow aggressive restarts.
The referred PR () only fixes the issue for kube-apiserver. We still need a generic fix in genericapiserver (Ref ).


Question: }) } expectedPageFaultsUpperBound := 1000000 <del> expectedMajorPageFaultsUpperBound := 10 </del> <ins> expectedMajorPageFaultsUpperBound := 15 </ins> if IsCgroup2UnifiedMode() { // On cgroupv2 these stats are recursive, so make sure they are at least like the value set // above for the container.

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --- ci-crio-cgroupv1-node-e2e-conformance many Many tests failing, they seem to be timing out. Example: Flakiness began late on 07/21 with this failure run: Note that cri-o jobs do not pin version of cri-o or Fedora CoreOS so it is possible the underlying components got bumped. /sig node
/cc
/milestone v1.22
I will take this /assign
/triage accepted
/remove-kind flake /kind failing-test as stats/summary test is now continuously failing
it is hard to tell because the tests were very flaky before, but I am suspicious of for causing the perma-failures. I have opened to test, where we can compare the difference in failures. It's possible this is just due to some crio configuration not being happy with the metrics server bump
/milestone v1.23
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /remove-kind failing-test /kind flake back to flaking:
it looks like the major page fault value is a bit off. I've opened to fix


Question: s := object.NewSearchIndex(client.Client) <del> var svm object.Reference for _, v := range addrs { ip, _, err := net.ParseCIDR(v.String()) if err != nil { return "", fmt.Errorf("unable to parse cidr from ip") } // Finds a virtual machine or host by IP address. svm, err = s.FindByIp(ctx, dc, ip.String(), true) if err == nil && svm != nil { break } } if svm == nil { return "", fmt.Errorf("unable to retrieve vm reference from vSphere") </del> <ins> svm, err := s.FindByUuid(ctx, dc, strings.ToLower(strings.TrimSpace(vmUUID)), true, nil) if err != nil { return "", err </ins> } var vm mo.VirtualMachine <del> err = s.Properties(ctx, svm.Reference(), []string{"name", "resourcePool"}, &vm) </del> <ins> err = s.Properties(ctx, svm.Reference(), []string{"name"}, &vm) </ins> if err != nil { return "", err } return vm.Name, nil }

What keywords did you search in Kubernetes issues before filing this one?: VMware, vSphere, IP, UUID BUG REPORT. Kubernetes version: Environment: Cloud provider or hardware configuration: VMware vSphere 6.5.0 OS : Debian GNU/Linux 8.7 (jessie) Kernel : Install tools: Kargo What happened: Some containers were launched on a node with a name containing the hostname of a different node and the does not retrieve the right node name : Here is retrieved instead of . When using the VSphere Cloud provider, the node name is retrieved from the function in Since this PR (), the node data are retrieved based on the IP address instead of the UUID. The function gets all the IP address and check if one of the IP on any interface matches a VM on vSphere. The problem is that the docker0 interface has the same IP on all the nodes (usually 172.17.0.1) and this IP is referenced into VMware. As this IP may be the first to appear in the it will match the first VM that vSphere gives with the function and this can be any VM running Docker on vSphere. Thus, the provider has a big chance to not get the informations for the VM it is running on, leading to an inconsistent state in the cluster (wrong pod name for example). Anything else we need to know: An idea would be to first retrieve the VM name with the function and if no results use the IP address (and add an exclusion to not use the interface). What do you think ? If you think this might be the right way to do, we can try to propose the PR. Thanks for your feedback.
Wow, this very much is a bad, bad bug. It was introduced by one of my colleagues here at HPE. It's made worse if you run multiple Kubernetes clusters in the same vSphere Datacenter. I'll send a PR upstream tomorrow to resolve this. I think it's best to go back to using dmidecode, but maybe someone from VMware ( or ?) has another suggestion? This would still require root access, but I can't think of a better way to get the machine's ID.
Related issue Will dig in more. For now instead of making it a value determined runtime the deployment framework should have root privileges and can populate it as part of the cloud config post VM creation.
is my colleague.

A little more info and a few ideas. I would also support going back to ` (still requires root though). We could also have the following logic: Get UUID and use it to get the VM Name with FindByUUID if possible, If not, get the hostname and use FindByDNSName, If still nothing is found, keep the current logic, excluding docker0 from the interfaces we examine. However, I agree with that getting this from the cloud config may be the best and most reliable solution.
I'm working on a patch right now that will do the following: Add a VMUUID field to the cloud config If VMUUID is not set, will use Error out if neither one is available
PR is - if someone would like to review :)


Question: // according to whatever system is managing the endpoint. A nil value // indicates an unknown state. In most cases consumers should interpret this // unknown state as ready. For compatibility reasons, ready should never be <del> // "true" for terminating endpoints. </del> <ins> // "true" for terminating endpoints, except when the normal readiness // behavior is being explicitly overridden, for example when the associated // Service has set the publishNotReadyAddresses flag. </ins> // +optional Ready *bool `json:"ready,omitempty" protobuf:"bytes,1,name=ready"`

The EndpointSlice API says and it works as expected in normal circumstances I delete the pod and the endpoints conditions are the expected However, if during this time I set we run into an incoherence with the API doc, but it seems that we already documented this edge case in the code I think that at this point is just to amend this information to the API, no? /kind bug /kind documentation
/sig network /assign
I would like to work on this. I just want a bit more clarification on what needs to be done!
I proposed a solution, that is clarify it in the doc ... once we reach an agreement in this issue we implement it ...
+1 to docs. I'd make it a little more vague, like: """ when the normal readiness behavior is being explicitly overridden, for example when the associated Service has set the publishNotReadyAddresses flag. """
so, the thing is to update the docs here with Tim's suggestion and here and do to update the generated code and submit a PR
/assign
I am not able to run command in windows, I am checking it. This error message is coming: *
/triage accepted
Is there a PR for this? Happy to review
I will raise the PR if my above problem gets solved. I am trying to make an update but the error mentioned above is coming.
What platform are you on? If the Makefile were really broken EVERYTHING would be failing.
he's on windows ... you will need a linux/macos. see
Let me know if you want someone to take it over.
I think in that case, you can assign this to someone else.


Question: }, expectedSlices: &slicesByAction{ toUpdate: []*discovery.EndpointSlice{ <del> simpleEndpointSlice("baz", "10.2.3.4", discovery.AddressTypeIPv4), </del> simpleEndpointSlice("bar", "10.1.2.3", discovery.AddressTypeIPv4), <ins> simpleEndpointSlice("baz", "10.2.3.4", discovery.AddressTypeIPv4), </ins> }, }, }, {

seen in the PR updating to go1.15, but this unit test should not be flaky /assign
was the flaky run
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
can reproduce locally with:
looks like toCreateByAddrType is a map which doesn't have a defined iteration order, so the address family order of toUpdate is not well-defined
/assign
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.


Question: ginkgo.It("should log default container if not specified", func(ctx context.Context) { ginkgo.By("Waiting for log generator to start.") <del> if !e2epod.CheckPodsRunningReadyOrSucceeded(ctx, c, ns, []string{podName}, framework.PodStartTimeout) { framework.Failf("Pod %s was not ready", podName) </del> <ins> // we need to wait for pod completion, to check the generated number of lines if err := e2epod.WaitForPodSuccessInNamespaceTimeout(ctx, c, podName, ns, framework.PodStartTimeout); err != nil { framework.Failf("Pod %s did not finish: %v", podName, err) </ins> } ginkgo.By("specified container log lines")

It seems that it was recently and it was always failing ! /sig cli /kind failing-test /assign
/triage accepted


Question: config, _, _ := volume.NewGlusterfsServer(c, namespace.Name) name := config.Prefix + "-server" defer func() { <del> volume.TestCleanup(f, config) </del> <ins> volume.TestServerCleanup(f, config) </ins> err := c.CoreV1().Endpoints(namespace.Name).Delete(context.TODO(), name, nil) framework.ExpectNoError(err, "defer: Gluster delete endpoints failed") }()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In block volume tests, wait for the Pod to be deleted instead of letting it be handled by the framework's namespace deletion Why is this needed: I think the regression caused by could have been caught if the tests check for Pod deletion because the Pod object shouldn't be deleted if TearDown fails.
/assign
This issue is not specific to block tests. We should in ideally wait for pod deletion at the end of all types of tests to confirm cleanup logic is functioning
I checked the tests under test/e2e/storage/testsuites/ where DeletePodWithWait is used to wait for the deletion.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten is this still an issue? One way to check is to make a breaking change in cleanup and see if the pull job fails.
I went through all block volume usage in e2e tests: All of them but one wait for their pods to be deleted. I'm fixing the remaining test in


Question: "io/ioutil" "net" "net/http" <del> "os/exec" </del> "path" "strconv" "strings"

Actually appending fqdn suffix already causes issue for non-resolvable node names based on today: Today when NodeController "registers" nodes in SyncCloud by getting bare instance names () from GCE, and recreating the fqdn and append it to instance names().


Question: "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/api/errors" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/fields" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/wait" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/watch" . "github.com/onsi/ginkgo"

This is wrong because something else could have updated the pod in the mean time, invalidating your resource version. (Specifically, kubelet writes statuses.) It should do a get/update loop that retries as long as it gets a conflict error. to find an owner
PR almost ready - just struggling to get e2e tests to run :-)
At last. SUCCESS! -- 5 Passed 34 Skipped I0424 09:04:47. ] All tests pass PR incoming in a few minutes.


Question: } // chooseHostInterfaceFromRoute cycles through each default route provided, looking for a <del> // global IP address from the interface for the route. addressFamilies determines whether it // prefers IPv4 or IPv6 </del> <ins> // global IP address from the interface for the route. If there are routes but no global // address is obtained from the interfaces, it checks if the loopback interface has a global address. // addressFamilies determines whether it prefers IPv4 or IPv6 </ins> func chooseHostInterfaceFromRoute(routes []Route, nw networkInterfacer, addressFamilies AddressFamilyPreference) (net.IP, error) { for _, family := range addressFamilies { klog.V(4).Infof("Looking for default routes with IPv%d addresses", uint(family))

What happened: When attempting to kubeadm join a control plane node or worker on a machine with global IPv4 unicast addresses on a loopback address (router to the host w/ BGP Unnumbered), the join fails with the following output: See this issue for details : Also: What you expected to happen: The correct IPv4 address should be selected from the loopback adapter, or we should be able to specify an IPv4 address to use for egress. The IPv6 addresses on the network adapters themselves are only used to establish upstream peering. IPv4 nexthops are fake 169.254.x.x addresses artificially programmed via RFC5549 into the ARP table. How to reproduce it (as minimally and precisely as possible): On a node with the following conditions: Routing to the host with FRR IPv6 link locals on physical network interfaces No IPv4 addresses on physical network interfaces Unicast IPv4 on the loopback adapter IPv4 default route present via RFC5549 Attempt to kubeadm join a node, and it will fail. Anything else we need to know?: We've been patching kubeadm with , which works as expected. There was a fix applied to allow initing control plane nodes w/ bind address 127.0.0.1 which would then select an IPv4 from the loopback, but it doesn't appear to have fixed the joining issues. Pinging and by request on kubeadm issues. Environment: Kubernetes version (use ): 1.17.12 Cloud provider or hardware configuration: baremetal OS (e.g: ): Centos 7 Kernel (e.g. ): 3.10.0-1127.19.1.el7.x86_64 Install tools: kubeadm Network plugin and version (if this is a network-related bug): Calico Others: FRR for peering with routers on physical network interfaces
/sig api-machinery
is defined but we should note the same function is used for the --bind-address of the kube-apiserver. /sig network
/assign
/triage accepted


Question: # Otherwise, we respect whatever set by the user. gci_images=( $(gcloud compute images list --project google-containers  --regexp='gci-dev.*' --format='value(name)') ) <del> if [[ -z "${KUBE_GCE_MASTER_IMAGE:-}" ]]; then </del> <ins> if [[ -z "${MASTER_IMAGE:-}" ]]; then </ins> MASTER_IMAGE="${gci_images[0]}" MASTER_IMAGE_PROJECT="google-containers" fi <del> if [[ -z "${KUBE_GCE_NODE_IMAGE:-}" ]]; then </del> <ins> if [[ -z "${NODE_IMAGE:-}" ]]; then </ins> NODE_IMAGE="${gci_images[0]}" NODE_IMAGE_PROJECT="google-containers" fi

We recently switched GKE's master node to GCI image, but not GCE master node. This discrepancy causes a lot of issues for us, especially some prs merged might work on GCE, but totally break GKE. We should get rid of that difference. cc/
/cc (two build cops who have recently suffered the effects of this).
The master image for test is defined here: We want to switch all test jobs? Or just one particular job, e.g., k8s ci testing k8s master head? I am afraid that simply replacing the image in that script will affect all test jobs
I think this is a bad plan unless we continue to test container VM start up somehow. We can't break that while people still use it!
An alternative: Set up another jenkins job dedicated for the purpose and mark it as commit queue blocker.
It may be better to just switch GKE masters back to containerVM for now, then redo the sequence (i.e. switch GCE to GCI, then GKE master to GCI).
if we switch both the test and non-test default from cvm -gci then starting with the 1.3 release people won't be using it (by default), so testing becomes less important.
I don't think we have to. The GKE CI job is already a CQ blocker, but it runs after submit so bad PRs still get merged (though they get identified quickly). What we need is a PR builder that runs tests with GCI, so we can block bad changes in the first place.
Definition of PR builers: cc/
Yes. It could be even simpler, it just has to start up a cluster and tear it down. Just run like 1 fast test.
We've got to do something though, the queue has been blocked for like 3 hours on GKE tests not passing.
I'd still like to run the e2e tests as some tests make assumptions of the OS distro and thus fail on GCI. Can we duplicate (that runs gce-e2e on every PR), replacing ContainerVM with GCI as the master and (or) node image?
It's a bit of a pain to do this, since that job also does a full build and it feels like a major waste to build each PR again. The Jenkins agents probably can't handle that much load, anyway.

at some point we probably need to bite the bullet and set up pipelines so we only build kubernetes once, rather than separately in the unit/integration, e2e, and node e2e jobs. I have no idea how well any of that interacts with the pull request plugin, of course.
Thanks for the information . So looks like the first alternative, running tests with GCI/GKE and block bad changes in the first place, is hard to get done due to some infrastructure challenges (as mentioned in ). It's not impossible though, just not prioritized right now. The second alternative would be, keep running the post-submit tests (the GKE e2e tests), and fix issues as they appear, which is not too bad imo since they run every 30 min and block submit queue on failure. At the end of the day, you cannot prevent 100% of breakages in presubmit tests, and that's why we have to have postsubmit tests to complement.
I think start another jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM is the simplest thing unblock us. The newly created job should only run the basic test suites. After 1.3, I suggested we do
No, this is not acceptable! we've been blocked for 10 out of the last 24 hours due to this! :)
I agree that we need to catch this before merging instead of after merging. The question is how best to do so.
The problem is that the submit-queue puts them in at odd hours, and we don't expect OSS oncall to be watching at 2 AM.
Well, the flip side is you don't have to run all tests (on all providers) so that all your PRs get merged faster. I am not saying I prefer postsubmit tests, it just something you have to face, no matter its gke tests or gce tests. I would prefer that too.
I think the solution to this is to run more tests when the PR is created, but then when the submit queue asks for a retest before merging only run a smoke test. I think that's a good tradeoff.
That's a good topic to talk and I have some ideas about that too, but I'd stop dragging the focus too far.

:smile:
I don't see any compelling reasons to have a different master image-type for GCE and GKE -- there are strong benefits to them being identical, as we've observed. We want 1.3+ to run GCI for GKE, therefore I propose we switch both GCE and GKE to use GCI for the master on all 1.3+ tests by default, including PR and post-submit. If there is a testing gap in that situation, because we think it's important to support containerVM for 1.3+ masters on GCE (but not GKE), then add coverage for that gap. I don't have enough context to know whether that's pre-submit or post-submit or both.
(To append: I think post-submit coverage is horrible and would support a P0 of any test tiger team and/or code-yellow exit criteria being that we have pre-submit coverage of all the various test matrix points. I realize that'll be expensive in both time and machine resources, but it'll be less expensive than our current approach and I suspect generate significant velocity benefits.)
This is a decent idea.
is out for vacation, could you please pick up this one? I think we agreed to have a separate jenkins job as one more pr builder, which have the master node on GCI image and rest of nodes on ContainerVM. The newly created job should only run the basic test suites. If you need some help, please ask
After talking with and we came to the following agreement: will switch the default base image for master nodes running on GCE from debian container-vm to GCI. will add a jenkins job that runs periodically (post-submit) that will continue to test master nodes running the debian container-vm, initially running against the master branch. will pin this job to the 1.3 release branch once it is cut, and no longer run it against the master branch. will delete the new jenkins job once we have full confidence that we will not roll back the master image from GCI to the debian container-vm, sometime between now and the 1.4 release.
I am going to work on a PR for no. 1. cc/
no. 2 is being in
The build broke again because of this:


Question: visibility = ["//visibility:public"], deps = [ "//pkg/kubelet/cm/cpuset:go_default_library", <ins> "//vendor/github.com/golang/glog:go_default_library", </ins> "//vendor/github.com/google/cadvisor/info/v1:go_default_library", ], )

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: CPU Manager uses topology from cAdvisor which uses which is different from in terms of coreID's. assigns coreID's in unique manner (platform-wise) whereas has coreID's from CPU registers which are unique only on socket level. This causes some unexpected cpu assignment on platforms with more than one socket. All tests written all "false positive" Example output: From What you expected to happen: Fix either by changing the way CPU manger sees topology (use ) or redesign CPU Manager to handle coreID's as socket only unique How to reproduce it (as minimally and precisely as possible): kubelet on dual socket node G pod that requests 2 cores pod gets 2 cores on different sockets
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node


Question: } handCoordinateMap[handCoordinate]++ } <ins> numHandsSeen := len(handCoordinateMap) </ins> <del> t.Logf("Deck size = %v, hand size = %v, number of possible hands = %d, number of hands seen = %d, number of deals = %d, expected count range = [%v, %v]", test.deckSize, test.handSize, allCoordinateCount, len(handCoordinateMap), test.hashMax, minCount, maxCount) </del> <ins> t.Logf("Deck size = %v, hand size = %v, number of possible hands = %d, number of hands seen = %d, number of deals = %d, expected count range = [%v, %v]", test.deckSize, test.handSize, allCoordinateCount, numHandsSeen, test.hashMax, minCount, maxCount) </ins> // histogram maps (count of times a hand is seen) to (number of hands having that count) histogram := make(map[int]int)

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which test(s) are flaking: Testgrid link: While there aren't any visible failures in the above testgrid link, we've observed flaky timeouts downstream. Reason for failure: Test times out Anything else we need to know: We don't have full logs for failed unit tests, so it's hard to pinpoint exactly which test is timing out, but I believe the offending function is this: You will see that the test iterates times: Summing up the hashMax for all four subtests, we see that the function iterates ~34 million times. While the test takes just ~9 seconds on my machine to complete, the CI is probably struggling with it (other tests are running in parallel and iterating 34 million times over a non-trivial loop is very CPU hungry). A successful execution of the unit test on CI takes ~3-4 minutes on average. The test doesn't have any of the usual suspects that can cause a timeout (like <-chan receive, i/o reads, wait.Poll(), time.Sleep(), etc.), so I suspect the large number of iterations to be the culprit. I propose making the unit test smaller and quicker to complete. Hopefully someone who understands shufflesharding better than me can take over :) /sig api-machinery /cc
when run with , the runtime increases significantly: that time is dominated by TestUniformDistribution
The test does a lot of slice appends. Maybe is constraining the allocator? There isn't much else in the test that might be slowed down by : no mutexes, no RNGs, no atomic operations, no channels, etc.
It looks like the buffer is pre-allocated to be the correct size though, so none of those appends should be resulting in allocations?
/triage accepted
BTW, in another branch I tried taking the s out of the test, instead calling with a lambda that writes into at a position selected by an advancing index. The got a little slower!


Question: var buffer bytes.Buffer buffer.WriteString("n") buffer.WriteString("# Entries added by HostAliases.n") <del> // write each IP/hostname pair as an entry into hosts file </del> <ins> // for each IP, write all aliases onto single line in hosts file </ins> for _, hostAlias := range hostAliases { <del> for _, hostname := range hostAlias.Hostnames { buffer.WriteString(fmt.Sprintf("%st%sn", hostAlias.IP, hostname)) } </del> <ins> buffer.WriteString(fmt.Sprintf("%st%sn", hostAlias.IP, strings.Join(hostAlias.Hostnames, "t"))) </ins> } return buffer.Bytes() }

What would you like to be : Currently, specifying multiple aliases for a single host using results in a line for each alias in . Example: leads to an file containing something like Why is this needed: Having multiple entries for the same IP address in can break DNS resolution for some software. See for more details <!-- DO NOT EDIT BELOW THIS LINE --/kind feature
/sig node


Question: function kube::build::clean_output() { # Clean out the output directory if it exists. <del> if kube::build::is_osx ; then if kube::build::build_image_built ; then echo "+++ Cleaning out boot2docker _output/" kube::build::run_build_command rm -rf "${REMOTE_OUTPUT_ROOT}" || true else echo "!!! Build image not built.  Assuming boot2docker _output is clean." fi </del> <ins> if kube::build::build_image_built ; then echo "+++ Cleaning out _output/ via docker build image" kube::build::run_build_command rm -rf '${REMOTE_OUTPUT_ROOT}/*' else echo "!!! Build image not built.  Cannot clean via docker build image." </ins> fi <del> echo "+++ Cleaning out _output directory" </del> <ins> echo "+++ Cleaning out local _output directory" </ins> rm -rf "${LOCAL_OUTPUT_ROOT}" }

Automation should work to unblock PRs that are not getting attention from reviewers. Nagging with ping messages Reassigning to a new reviewer (leave a comment so we can track when this is happening and why) Flagging the PR as needing manual intervention from an on call role Allow reviewers to leave vacation times so they don't get assigned Tracked in:
handing off to you. is now merged.
1737 adds labels to denote the state of the PR: "state/prereview", "state/needs-review", "state/needs-changes", "state/needs-approval"
Issues go stale after 30d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: // TODO: Get rid of [DisabledForLargeClusters] tag when issue #56138 is fixed. It("should be able to switch session affinity for LoadBalancer service with ESIPP off [Slow] [DisabledForLargeClusters]", func() { <ins> // L4 load balancer affinity `ClientIP` is not supported on AWS ELB. framework.SkipIfProviderIs("aws") </ins> svc := getServeHostnameService("service") svc.Spec.Type = v1.ServiceTypeLoadBalancer svc.Spec.ExternalTrafficPolicy = v1.ServiceExternalTrafficPolicyTypeCluster

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: The test should either succeed or be skipped on AWS How to reproduce it (as minimally and precisely as possible): Run this test on an AWS cluster with the AWS cloud provider configured Anything else we need to know?: The test sets the service's session affinity to here: The AWS cloud provider specifically only supports : Environment: Kubernetes version (use ): Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
cc
/cc
Thanks for the investigation, given the information so far, we should just skip this test on AWS. Will prepare a PR soon.


Question: } glog.Infof("Found tombstone for %v", key) needUpdate = cc.processServiceDeletion(cachedService, clusterName) <ins> isDeletion = true </ins> } } else { glog.Infof("Can not get service %v for cluster %s from serviceStore", key, clusterName) needUpdate = cc.processServiceDeletion(cachedService, clusterName) <ins> isDeletion = true </ins> } if needUpdate {

As mentioned in when the service is deleted from the underlying cluster, the controller should recreate it, but it does not. Below's a transcript. is talking to the federation-apiserver, and is talking to the normal cluster it's pointed at.
Thanks for catching this cc
cc
This is definitely a bug, but I think that we can kick it out into v1.4. Until fixed (i.e. while in beta), we can tell customers not to delete the underlying services in their clusters.
it should not be, I have tested this before. How long did you wait when you check the service existence in last step? Can you send me the logs? The service deletion will cause federation service update, and then the federation service watcher will catch the update even and call another round of service updates to all clusters. When you delete the service, if the service got ingress ip already? I am thinking if the deletion did not trigger federation service update? So so no update event generated..
I think that the best way forward will be to encode this in a reproducible e2e test, otherwise there are too many variables to debug effectively. It's great to know that you've tested this case, and it works in at least some scenarios.
one possible solution come into my mind, currently we set the service full sync time to 10 minutes , if we change it back to 30s, then at least during the full sync, the deleted service will be re-created. I will send a PR anyway, you can judge if it should be put to v1.3
I think we need to find and fix the underlying problem, not cover it up with a shorter sync period. If we have the correct watch on each service, we will be notified as soon as they are deleted. Why is the watch notification not firing correctly?
Keeping this open until we've verified this fix in e2e tests.
I believe this to be fixed, but we still lack an e2e test to verify conclusively. would you like to take this on along with the other e2e tests that you are busy adding?
ping
plan on doing this for 1.4? or bump to 1.5?
Yes, 1.5 is fine.
switching this to 1.6 as it's too late for 1.5. ok?

Moving to 1.7 as late to happen in 1.6. Feel free to switch back if this is incorrect.


Question: "strings" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/cluster-bootstrap/token/api" legacyutil "k8s.io/cluster-bootstrap/token/util"

What happened: kube-controller-manager with enabled option --controllers=,tokencleaner is not removing newly created tokens. It works only after kube-controller-manager restart or after editing secret with token. What you expected to happen: It should remove all expired tokens from kube-system namespace, without restarting or touching secrets. How to reproduce it (as minimally and precisely as possible): enable tokencleaner in kube-controller-manager start controller manager create bootstrap token, you can use script wait 5 minutes token is still there restart kube-controller-manager OR edit newly created token token should be gone after a while Anything else we need to know?: It could be related with Environment: Kubernetes version (use ): Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-12-03T20:56:12Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Do you see the following log ? thanks
no errors at all
/sig auth


Question: Expect(err).NotTo(HaveOccurred()) firstCond := client.ReplicaSetHasDesiredReplicas(c.Extensions(), first) <del> wait.PollImmediate(10*time.Millisecond, 1*time.Minute, firstCond) </del> <ins> err = wait.PollImmediate(10*time.Millisecond, 1*time.Minute, firstCond) Expect(err).NotTo(HaveOccurred()) </ins> secondCond := client.ReplicaSetHasDesiredReplicas(c.Extensions(), second) <del> wait.PollImmediate(10*time.Millisecond, 1*time.Minute, secondCond) </del> <ins> err = wait.PollImmediate(10*time.Millisecond, 1*time.Minute, secondCond) Expect(err).NotTo(HaveOccurred()) </ins> By(fmt.Sprintf("Updating the size (up) and template at the same time for deployment %q", deploymentName)) newReplicas := int32(20)

https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in . Previous issues for this test:
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s-

Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes

e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite} Happened on a presubmit run in .
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}
https://k8s- Failed: [] Deployment scaled rollout deployment should not block on annotation check {Kubernetes e2e suite}


Question: framework.ExpectEqual(testRcPatched.ObjectMeta.Labels["test-rc"], "patched", "failed to patch RC") ginkgo.By("waiting for RC to be modified") eventFound = false <del> ctx, cancel = context.WithTimeout(context.Background(), 30*time.Second) </del> <ins> ctx, cancel = context.WithTimeout(context.Background(), 60*time.Second) </ins> defer cancel() _, err = watchUntilWithoutRetry(ctx, retryWatcher, func(watchEvent watch.Event) (bool, error) { if watchEvent.Type != watch.Modified {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: - Triage link for more jobs that are affected: Which test(s) are failing: Since when has it been failing: Since October 23 Testgrid link: Reason for failure: Anything else we need to know: Example spyglass link: /sig apps /cc /priority important-soon
/cc missed this in the main body
/assign /triage accepted
/remove-priority important-soon /priority critical-urgent top flake for pull-kubernetes-conformance-kind-ipv6-parallel, 2nd top flake for per /area conformance this is a conformance test /remove-kind failing-test /kind flake this is not failing continuously, but is instead flaking
/assign
I increased the timeout while waiting for available Replicas.
Test grid green since Merge 5 Nov 2020 /cc


Question: "k8s.io/apiserver/pkg/authentication/authenticator" "k8s.io/apiserver/pkg/authentication/user" <ins> "k8s.io/apiserver/pkg/endpoints/request" </ins> "k8s.io/client-go/rest" )

Semantics are going to vary. We need to cross off these authenticators: [x] if ctx has an audience limitation, verify the token audience and echo in the response [x] take implicit audiences at WebhookTokenAuthenticator construction time if ctx is audience-limited, add into the tokenreview object if the tokenreview returns with an audience set that matches the ctx, copy into the response and return success if the tokenreview returns without an audience set, ensure the ctx audiences intersect with the implicit audiences taken at WebhookTokenAuthenticator construction time, and pick one to add into the response [x] echo audience in response if present in ctx [x] echo audience in response if present in ctx [x] check that the the order of (audience in the context) ∩ ( (implicit api audiences) ∪ (clientids) ) 0 Take implicit audiences at Authenticator construction time, and if ctx is audience-limited, ensure it intersects with the implicit audiences, and echo an audience in the response. [x] [x] [x] [x] [x] [x] Do nothing? [x] [x] [x] [x] [x] [x] [x] note: we already keyed the cache with the audience. [x]
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: if len(file.Path) == 0 { allErrs = append(allErrs, field.Required(fldPath.Child("path"), "")) } <del> allErrs = append(allErrs, validateLocalNonReservedPath(file.Path, fldPath.Child("path"))...) </del> <ins> allErrs = append(allErrs, ValidateLocalNonReservedPath(file.Path, fldPath.Child("path"))...) </ins> if file.FieldRef != nil { allErrs = append(allErrs, validateObjectFieldSelector(file.FieldRef, &validVolumeDownwardAPIFieldPathExpressions, fldPath.Child("fieldRef"))...) if file.ResourceFieldRef != nil {

Originally posted by in
Hello, I'm new here. Can I try solving this? it seems like a good first issue
sure, context is the change is needed here: We have to tolerate existing pods / pod specs that fail this validation, so we should only do this additional validation on create or if the existing object passes this validation. To do this: an field to , initialize that field to false, and then conditionally override it to if is non-nil and has a projected token volume path that fails (will have to export the function to check this) , if is false, call the validation on the path:
Hi, I'd like to work on this if there is no one is proceeding on. /assign


Question: kube::golang::setup_env <ins> # The sort at the end makes sure we feed the topological sort a deterministic # list (since there aren't many dependencies). </ins> generated_files=($( find . -not (  (

See, e.g. This seems to be blocking multiple PRs from even being tested.
Looking.
I think I figured it out. The order of generation matters. The verify script subtly breaks that by renaming and checking the files one at a time. The update script deletes everything and then recreates them.
I have a fix. Will send a PR shortly.


Question: "k8s.io/kubernetes/pkg/features" kubeletconfig "k8s.io/kubernetes/pkg/kubelet/apis/config" kubelettypes "k8s.io/kubernetes/pkg/kubelet/types" <ins> e2eskipper "k8s.io/kubernetes/test/e2e/framework/skipper" </ins> testutils "k8s.io/kubernetes/test/utils" )

Failing on cri-o serial job. Looks like the dbus command isn't available for graceful shutdown tests that aren't running on Ubuntu. /assign /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
/triage accepted /priority important-soon
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
hmm, is this still issue? The testgrid job is still failing even after the fix appears to be merged? !
Please take a look when you get a chance, the error still seems to be that dbus-send is unavailable... /cc
oops... I forgot to add this to the other


Question: embed = [":go_default_library"], deps = [ "//pkg/kubelet/apis/pluginregistration/v1:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", </ins> "//vendor/github.com/stretchr/testify/require:go_default_library", "//vendor/k8s.io/klog:go_default_library", ],

What happened: A race condition in the plugin watcher might happen when a is created, started and immediately stopped and cleaned up. If executed several times, the unit test will expose this issue. The PR addressed part of the problem by serializing the handling of CREATE and DELETE events. However, a race might still happen in the function. describes in detail the problem. How to reproduce it (as minimally and precisely as possible): /sig storage /sig node
/assign


Question: } ps.Containers = append(ps.Containers, *cs) } <ins> // cleanup outdated caches. p.cleanupOutdatedCaches() </ins> result := make([]statsapi.PodStats, 0, len(sandboxIDToPodStats)) for _, s := range sandboxIDToPodStats {

Related: On Windows, kubelet reports only the cumulative for cpu usage. On Linux, kubelet reports both and the instantaneous . Although In the future, most components/monitoring pipelines will switch to using the cumulative metrics, a few features still depend on the instantaneous metrics today. Examples are: , HPA/VPA. Kubelet should report this metrics on Windows to support the existing features. /cc who explained to me what features depend on the cpu metrics. /cc /cc
- is this something that's clear enough in the CRI layer that you can implement it, or do we need to fix that first?
/assign
do you have any notes or a doc you can point to that better describes what features depend on which metrics? It would be good for us to share and see if anything other than is broken
The horizontal and vertical pod autoscalers also depend on the
Documentation on metrics API and metrics server (i.e., the aggregator): The metrics-server pod keeps spewing errors like this: Running : I haven't verified whether pod autoscaling works in my cluster, but I assume they won't since metrics server seems to discard the metrics from the windows pods. I believe the implementation is in kubelet (for the docker integration), so this should not need CRI changes.
I think this is generally an issue for non-Docker CRIs (not necessarily Windows-exclusive). The kubelet source code suggests that these metrics will be gathered from the CRI (if the CRI is not Docker). The struct that kubelet expects returned from the ContainerStats RPC does not include usageNanoCores. Does this mean that, until this is resolved, custom CRIs are unable to satisfactorily report metrics to the 'metrics-server'?
- are you able to work on this?
that is a good point. For CRI runtimes, the kubelet should calculate the average over 10s to produce this metric.
Good point. We designed CRI around core metrics API, but some CRI runtimes have integrated with cadvisor directly to fill the gap while we wait for core metrics API. I think it's reasonable for kubelet to do this on behalf of all CRI runtimes. I spoke to offline, and this would probably require no more than 2 days of work. If sig-windows (e.g., can pick this up, that'd be great.

has agreed to review the changes (thanks!) If not, we should clearly document that this is not supported in Windows GA for 1.14.
thanks for the update. we will wait to hear back from
Currently, is got from docker's stats API on Windows. From docker stats API , only is available, hence is not set on Windows: Do you know any way to get instantaneous for Windows containers?
I have no idea. any idea?
I think this is broader than just a Windows issue. Officially supported tools such as the kubernetes metrics-server expect the field, but custom CRIs that conform to the interfaces defined in the package are not given the liberty to return this value.
suggested previously to cache the stats temporarily and compute the stats using the diff. David, could you chime in on how this could be done? Yes, this was mentioned in the previous comments too. There are plans to transition those components off this field, but it's WIP. We are looking for a workaround in kubelet for now.
my biggest worry is that if we say this impacts the horizonal POD autoscaler, how is that passing in our end2end and conformance tests?
Do you test HPA? It's not part of conformance, and I doubt this is included in windows testing.
As to how this could be done, in and we could just add a in the struct, which we assign the previous CpuUsage metric for each container. Then, when we need to calculate the rate, we pass the previous CpuUsage along with the current one to , and calculate a rate based on the timestamp and values.
ListPodCPUAndMemoryStats() we could just add a map[string]*runtimeapi.CpuUsage in the criStatsProvider struct I've done some tinkering on this myself and I originally thought that it would be as simple as that. It gets much more complicated when you have to work in between the existing cAdvisor-based logic. I assume the intent is to cut it out entirely eventually, but the information passed through the CRI doesn't even come close to being so comprehensive. So far I've found that this approach fixes the problem for the "/stats/summary" api, but not necessarily for the other apis. In particular, "/stats/container" seems to be completely based on cAdvisor, being unimpacted by the changes we made to those two functions.

Some monitoring tools and metrics scrapers rely on the "/stats/container" api for their metrics, which means it is insufficient to update one api without doing the other.
We don't need to fill in everything in the summary API, just "usageNanoCores". The "/stats/summary" api is the one used by the metrics server, which is why we care about it in this case. The "/stats/container" isn't something we need to implement to support this.
we a couple of tickets to validate if HPA for Windows containers works. My guess is that it works for memory and not for CPU metrics because of this issue. If this is complicated to fix, i wonder if waiting for the CRI containerD changes in windows would be the best option for supporting this.
containerd and cri-o bypass this issue by leveraging cadvisor as a temporary solution. I don't think that'd work for windows.
Sorry the for late response. HPA is actually working for Windows containers. e.g. in my cluster (v1.12.5), cpu-based HPA is working as expected: Which version of kubernetes are you referring for HPA issues?
are you running heapster, or the metrics server to serve HPA? Heapster uses the cumulative metric, but the metrics-server uses the 10 average rate metric.
metrics-server is used in my cluster.
See .
/milestone v1.14


Question: # Launches an nginx container and verifies it can be reached. Assumes that # we're being called by hack/e2e-test.sh (we use some env vars it sets up). <ins> # TODO: remove this once test/e2e/kubectl.go is green in jenkins </ins> set -o errexit set -o nounset set -o pipefail

Part of issue Re-write test in Go, generate unique names for top level resources, ensure test is idempotent, clean up all resources at end of test.
I was just glancing at this test to see how crunchy it was to port. One of the sad things would be abandoning , so if we do port this, I think we should definitely go through the to make sure it stays tested.
That reminds me... Do we have an issue yet to add an e2e test to specifically test kubectl?
We do now:
Thanks.
I am unassigning myself from this for now. I expect to be working on the fixit for cluster level logging for several days so I'd rather not have this assigned to me while I can't give it attention. If it is still open later then I may try to take a crack at it.