Question: func (f *KubeletFlags) addOSFlags(fs *pflag.FlagSet) { fs.BoolVar(&f.WindowsService, "windows-service", f.WindowsService, "Enable Windows Service Control Manager API integration") <ins> // The default priority class associated with any process in Windows is NORMAL_PRIORITY_CLASS. Keeping it as is // to maintain backwards compatibility. // Source: https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities fs.StringVar(&f.WindowsPriorityClass, "windows-priorityclass", "NORMAL_PRIORITY_CLASS", "Set the PriorityClass associated with kubelet process, the default ones are available at "+ "https://docs.microsoft.com/en-us/windows/win32/procthread/scheduling-priorities") </ins> }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Windows nodes running with high (90%) CPU usage are having calls to /stats/summary time out with "Context Deadline Exceeded". When this happens metrics-server cannot gather accurate CPU/memory usage for the node or pods running on the node which results in HPA not functioning returning <unknowninstead of values What you expected to happen: Windows nodes should be able to run with high CPU utilization without negatively impacting Kubernetes functionality How to reproduce it (as minimally and precisely as possible): a cluster with Windows Server 2019 nodes CPU intensive workloads into cluster ex: workload to put node into high CPU state node metrics with Anything else we need to know?: This issue appears to be easier to reproduce on 1.18.x nodes vs 1.16.x or 1.17.x nodes Ex: ! ! metrics-server logs show E1020 10:58:24. 1 ] unable to fully collect metrics: unable to fully scrape metrics from source kubeletsummary:akswin000000: unable to fetch metrics from Kubelet akswin000000 (10.240.0.115): Get https://10.240.0.115:/stats/summary?onlycpuandmemory=true: context deadline exceeded Kubelet logs show and Environment: Kubernetes version (use ): 1.18.8 Cloud provider or hardware configuration: AKS / Azure OS (e.g: ): WIndows Server 2019 Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: Docker EE version 19.3.11
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
/sig windows /area kubelet
I was able to reproduce on 1.18.8 during a scale operation: where it fully subscribes all CPU's in Allocatable: Metric server logs: I do not see "context deadline" in kubelet but I did find some logs assocated: Once the pods are scheduled the node remains at 100% cpu and I no longer see errors but is not accurate: on the node (run after the pod deployment stabilized)
Adding to kubelete's flag and I no longer see the errors. This reserves some cpu for kubelet and other processes. We call this out in the but do not do the same for cpu.
Also, without setting system reserved CPU, it looks like does not accurately reflect CPU usage on the nodes when CPU utilization is very high. When I got some nodes into this state I did not see any error messages in logs from metrics-server and also I noticed that that querying the metrics API directly from showed recent timestamps.
I run into this again during scaling, collectiong logs for analysis here: Metric Server logs: Kubelet logs:
have you experienced this at all?
I believe has seen high CPU issues before. Not particularly on this. I've heard there were kernel issues in the past that triggered high CPU cases.
Some investigations on our end are showing that the DiagTrack service running in the infra/pause container can consume non-trivial amounts of CPU. I'll create an issue to address that problem as it might help out some here. That certainly will not eliminate the problem we are seeing tho.
is an issue for DiagTrack
The differences in behavior between 1.17. and 1.18. clusters looks like it is caused by (which was NOT backported to any prior releases) K8s version <= 1.17 behavior Prior to these changes CPU weights for containers were inferred based on limit values set in the pod specs. The observed behavior on Windows system when CPU weights were used was that under high-CPU load pods/containers were not guaranteed their scheduling time. The good news here is that system critical services like kubelet/docker/etc were still able to get enough CPU time for the nodes to function properly. K8s version = 1.18 behavior After these changes CPU limits were enforced for pods.

The observed behavior on 1.18 clusters is the when a Windows node enters a high-CPU state is that containers are guaranteed their scheduling time but host resources start to get starved. Usually this manifests as the node entering the state, metrics not working correctly (because calls to /stats/summary are timing out) or both. Suggestion / possible fixes Set the Windows process priority of the kubelet to  to help ensure it gets CPU time Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) Set reserve CPU with with --system-reserves=[cpu=500m] This needs to scaled based on number of CPUs on host based on my experimentations Currently guidance on for Windows only mentions reserving system-memory It is still possible to over-commit CPU and starve kubelet/docker/HCS if users schedule pods with mis-matching requests and limits Update documentation.
Kubelet can do this itself though calls (GetCurrentProcess() and Get/SetPriorityClass()) I can work on this, I am wondering if we should do the same for CRI(like docker) and CNI processes.
Sure, if you want to work on this go ahead. I think we should probably try to make this be toggleable via a kubelet flag. I believe there are other windows specific flags already implemented in the kubelet (like one to run as a windows service IIRC)
docker/containerd should probably have a flag that allows them to run as high priory as well but would likely be different issues in the respective projects
/reopen I don't think this was ever fully addressed
Reopened this issue. <detailsIn response to : /reopen I don't think this was ever fully addressed Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
also related to
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: failureThreshold: 5 readinessProbe: httpGet: <del> path: /health port: 8080 </del> <ins> path: /ready port: 8181 </ins> scheme: HTTP securityContext: allowPrivilegeEscalation: false

Which jobs are failing: Multiple jobs on . Which test(s) are failing: on the following jobs: Since when has it been failing: Since 5/17. Testgrid links: See above. Reason for failure: All the failures mentioned above seem to have to have the same root cause, namely failure while waiting for CoreDNS to be ready. Example: /sig testing /priority critical-urgent /kind failing test /milestone v1.15 /cc
Some of this (possible all of it) is due to , which is dependent on being merged.
/sig cluster-lifecycle
Not the gce jobs thoght. Only the kinder job. So the gce jobs are failing with a problem outside of kubeadm/kinder but possibly still related to the corefile plugin change
/assign (At kubecon this week)
The gce failures appear to show the same behavior, but with kube-up instead of kubeadm. I.e. it appears that kube-up may also be retaining old configs when upgrading/downgrading. If thats true, then we may need to integrate the migration tools into kube-up as well.
I have opened which aims to fix all the failing tests except , which will be fixed via
Great job so far and Would it be possible to ask for the open PRs to be merged and/or a solution found soon? We have these tests failing for quite some time and code freeze coming up in the next days. Thank you in advance for your efforts!
thanks for the heads up. let me know if you don't have the time for the refactor that i requested and i can try to take over there. the alternative is to rollback the coredns version which i assume is not ideal, due to a variety of fixes in the new version.
is there any more work planned for this issue? I see that some of the kinder jobs are still failing after merged:
merging and will resolve and close this issue.
Thank you for the update !
updates: kinder tests were fixed via
/unassign i was here for: it's a bit flaky (which is due to the tooling), but green.
Since and we're moved to 1.16, and we're waiting for to merge, can we move this issue to 1.16?
? is that the right PR?

Good catch, I meant
The GCE test failures were fixed by and /close
Closing this issue. <detailsIn response to : The GCE test failures were fixed by and /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: } item, ok := f.items[id] if !ok { <del> // Item may have been deleted subsequently. </del> <ins> // This should never happen klog.Errorf("Inconceivable! %q was in f.queue but not f.items; ignoring.", id) </ins> continue } delete(f.items, id)

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: In reviewing another PR, I noticed incorrect commentary and code. The primary offense is at , where a comment asserts (incorrectly) that a key can be present in but not and the code removes a key from without ensuring it is absent from . The only saving grace is that this code is actually unreachable (as also noted in the comment). I think it would be better to fix the comment and in this unreachable situation, than to violate an invariant that is critical to other code in this file. also contemplates the possibility that a key can be in and absent from . What you expected to happen: I expected there to be no comments or code that contemplate violations of the invariant at How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): current branch Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
Sorry, I should have noticed this in my earlier pass over this code.
Agreed. Was discussing this with at
We discussed this one, Mike will send PR, Daniel will review.


Question: // Do not return error if the container does not exist if !libdocker.IsContainerNotFoundError(err) { errList = append(errList, err) <ins> } else { // remove the checkpoint for any sandbox that is not found in the runtime ds.checkpointHandler.RemoveCheckpoint(podSandboxID) </ins> } } return utilerrors.NewAggregate(errList)

/kind bug What happened: the kubelet keeps attempting to gc containers from dockershim checkpoint files. if the container is no longer present, we keep attempting to gc it because the file is never removed from checkpoint. What you expected to happen: checkpoint should not leak.
/sig node


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" pkgruntime "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/util/runtime" <ins> "k8s.io/apimachinery/pkg/util/sets" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/apimachinery/pkg/watch" "k8s.io/client-go/tools/cache"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"4", GitVersion:"v1.4.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-10T18:19:49Z", GoVersion:"go1.7.1", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.9", GitCommit:"", GitTreeState:"clean", BuildDate:"2016-10-20T20:37:18Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Local: El Capitan, Remote: GCI Kernel (e.g. ): Local: Darwin ... 15.6.0 Darwin Kernel Version 15.6.0: Thu Sep 1 15:01:16 PDT 2016; root:xnu-3248.60.11~2/RELEASEX8664 x86_64 Install tools: gcloud Others: n/a What happened: Provisioned 3 clusters each in different regions (us-central1, asia-east1, europe-west1). Each cluster is in 2 zones (w/ gcloud --zone... --additional-zones...) Provisioned Federation API Server and Controller (1.4.4) based on Verified clusters were ready. Provisioned federated service & deployed backends into each of the cluster. Verified the federated services were created, and all of the regional LB were created, and verified all the LB IPs were working w/ direct requests. DNS provider updated DNS w/ most of the service/zone CNAME records. But A records did not complete. Further investigation for the log points to multiple entries in rrdata w/ the same IP due to the multi-zone deployment: I1029 19:24:24. 1 ] Have recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180.

Need recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180I1029 19:24:24. 1 ] Existing recordset <(clouddns) "buttonmasher-" type=CNAME rrdatas=["buttonmasher-"] ttl=180is not equivalent to needed recordset <(clouddns) "buttonmasher-" type=A rrdatas=["130.211.203.196" "130.211.203.196"] ttl=180, removing existing and adding needed. After creating the clusters w/ only one zone, the DNS provider worked properly. What you expected to happen: Cloud DNS Provider should update the DNS record properly w/ multi-zone clusters. How to reproduce it (as minimally and precisely as possible): 3 x multi-zone clusters Federation API Server and Controller clusters in to federation federated service and backends Anything else do we need to know: Using GKE and Cloud DNS
/cc
Also hit this. Any workarounds?
Sorry, no direct workarounds available yet. You could convert your multi-zone clusters into multiple single-zone clusters, and federate all of those.
FYI, although I'm sure you're aware - also affects AWS multi AZ clusters (1.6.2)
This seems bad. I'll try to take a look into it.
Fixed by


Question: return &priorityMetadata{ podLimits:               getResourceLimits(pod), podSelector:             getSelector(pod, pmf.serviceLister, pmf.controllerLister, pmf.replicaSetLister, pmf.statefulSetLister), <del> controllerRef:           metav1.GetControllerOf(pod), </del> podFirstServiceSelector: getFirstServiceSelector(pod, pmf.serviceLister), podTopologySpreadMap:    tpSpreadMap, }

/sig scheduling /priority important-soon See as an example. Part of This shouldn't need a PostFilter Plugin, it is not worth caching the result of metav1.GetControllerOf. /assign


Question: BearerToken:      stringptr(""), Impersonate:      stringptr(""), ImpersonateGroup: &impersonateGroup, <ins> usePersistentConfig: usePersistentConfig, </ins> } }

/kind bug What happened: takes ~11 seconds when there are thousands of CRDs and is around 28KiB. logs the following message once per resource: Removing unnecessary entries from reduces the runtime by around ten seconds. What you expected to happen: only parses once, and therefore doesn't run unnecessarily slowly. How to reproduce it (as minimally and precisely as possible): shows the config being loaded multiple times (assuming you have multiple pods in the current cluster). Alternatively, add 10-15 clusters to and create ~3000 of a cheap resource (eg custom resources or empty config maps). Run on this resource and observe that it is unnecessarily slow. Anything else we need to know?: I've created a CPU profile using the latest CI build of kubectl, which shows ~12 seconds being spent in . Environment: Kubernetes version (use ): v1.12.1 (also Cloud provider or hardware configuration: GKE or minikube --vm-driver=none OS (e.g. from /etc/os-release): Debian Buster-like Kernel (e.g. ): 4.17.0
/sig cli
/assign
Same problem here. I have less resources but more contexts in my kubeconfig file. With 90 pods and ~300 contexts in the kubeconfig it also takes ~10 seconds for a simple kubectl get pods Note: I tried it with kubectl: 1.11.0, 1.12. With kubectl 1.10.5 the kubeconfig was loaded only once.
I'm unable to reproduce this as of the latest master. My kubeconfig is only loaded 4 times, no matter how many resources are loaded.
When I try a kubectl v1.13.0-alpha.3 against unfortunately a 1.9.11 Kubernetes Cluster I still have the problem: `
Maybe the bug is Linux-only? Presumably the kubeconfig location is different on different platforms, so maybe it has a different code path. I can repro on latest master:
I filed to fixed this. You can try to help validate it. Thanks.
Works for me, thanks!


Question: package cache import ( <ins> "fmt" </ins> "reflect" "testing"

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently some unit tests in pkg/scheduler is not organized as Use subtests can make it easier to find out which case fails and leverage benefit of parallelism . Why is this needed: Follow Go test practice.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-kind feature /kind cleanup /sig scheduling /assign


Question: ENV DO_NOT_INSTALL_CATCH_ALL_CONFIG true RUN apt-get -q update &&  <del> apt-get install -y curl &&  apt-get install -y gcc &&  apt-get install -y make &&  </del> <ins> apt-get install -y curl ca-certificates gcc make bash &&  apt-get install -y --reinstall lsb-base lsb-release &&  echo "Installing logging agent" &&  curl -sSL https://dl.google.com/cloudagents/install-logging-agent.sh | bash &&  /usr/sbin/google-fluentd-gem install fluent-plugin-record-reformer -v 0.8.1  &&  /usr/sbin/google-fluentd-gem install fluent-plugin-systemd -v 0.0.3 &&  apt-get remove -y gcc make &&  apt-get autoremove -y &&  </ins> apt-get clean &&  <del> rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* &&  curl -s https://dl.google.com/cloudagents/install-logging-agent.sh | bash # Install the record reformer and systemd plugins. RUN /usr/sbin/google-fluentd-gem install fluent-plugin-record-reformer -v 0.8.1 RUN /usr/sbin/google-fluentd-gem install fluent-plugin-systemd -v 0.0.3 # Remove the misleading log file that gets generated when the agent is installed RUN rm -rf /var/log/google-fluentd </del> <ins> rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*  /opt/google-fluentd/embedded/share/doc  /opt/google-fluentd/embedded/share/gtk-doc  /opt/google-fluentd/embedded/lib/postgresql  /opt/google-fluentd/embedded/bin/postgres  /opt/google-fluentd/embedded/share/postgresql  /var/log/google-fluentd </ins> # Copy the Fluentd configuration files for logging Docker container logs. # Either configuration file can be used by specifying `-c <file>` as a command

It's 400+mb and we need to pull it on every node. This leads to a lot of image pulling flake (), and also gets very expensive in some countries where bandwidth is sold at a premium. We took a stab at halving the size with: , but then realized that we were using an unsupported base image We tried to optimize to ~300Mb by building a smaller ubuntu:15 base, but that's still huge. Every other mandatory system daemon is built on busybox and so takes ~10Mb.
I can try again now that the agent supports
Newer Docker images using the spec from the official repository are based on AlpineLinux and are around ~38MB:


Question: return err } } <del> return nil </del> <ins> return manageJobErr </ins> } // pastActiveDeadline checks if job has ActiveDeadlineSeconds field set and if it is exceeded.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--What keywords did you search in Kubernetes issues before filing this one? Job, resourcequota, scheduler Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-03T20:44:38Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-04-14T13:36:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: CentOS bare metal () Cloud provider or hardware configuration: - OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): Linux <server name3.10.0-514.2.2.el7.x8664 SMP Tue Dec 6 23:06:41 UTC 2016 x8664 x8664 x8664 GNU/Linux Install tools: - Others: - What happened: Having created a resourcequota (e.g. 400M) and creating enough memory-limited jobs to go over that quota (e.g. 4 x 200M limit jobs), at first, two jobs are created and executed as expected, and two jobs fail at first due to memory quota: Events: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 4s 4s 1 job-controller Warning FailedCreate Error creating: pods "memhog-y152c55u-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi After the first two jobs have been completed, instead of starting the remaining jobs as soon as possible, the remaining two jobs stay unstarted in this state for many, many hours until they finally start and complete.

First job: Start Time: Fri, 28 Apr 2017 15:00:08 +0300 One of the remaining 2 jobs: Start Time: Sat, 29 Apr 2017 07:13:34 +0300 What you expected to happen: I'm expecting the two waiting jobs to start executing as soon as the resources they need are available, e.g. as soon as the first jobs are completed. How to reproduce it (as minimally and precisely as possible): Bring up a kubernetes cluster, then execute these commands: $ kubectl create quota memoryquota --hard=memory=400Mi (cat /dev/urandom head -n 1)' --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory='2'm run- $ chmod +x run- $ for i in {1.4}; do ./run- 200 100; done job "memhog-0ju2eiuy" created job "memhog-t8qyt5sj" created job "memhog-ljaqz4ek" created job "memhog-eb8q8kr3" created Wait for around 30 seconds and the first jobs should be complete: $ kubectl get jobs NAME DESIRED SUCCESSFUL AGE memhog-0ju2eiuy 1 1 43s memhog-eb8q8kr3 1 0 42s memhog-ljaqz4ek 1 0 43s memhog-t8qyt5sj 1 1 43s $ kubectl describe jobs . However, the remaining jobs don't start at all as shown by: $ kubectl describe job memhog-ljaqz4ek Name: memhog-ljaqz4ek Namespace: test Selector: controller-uid=-2f00-11e7-8766- Labels: run=memhog-ljaqz4ek Annotations: run memhog-ljaqz4ek memhog -r100 100m --replicas=1 --restart=OnFailure --image-pull-policy=IfNotPresent --limits=memory=200Mi,cpu=100m --record=true --image=derekway.

Parallelism: 1 Completions: 1 Pods Statuses: 0 Running / 0 Succeeded / 0 Failed Pod Template: Labels: controller-uid=-2f00-11e7-8766- job-name=memhog-ljaqz4ek run=memhog-ljaqz4ek Containers: memhog-ljaqz4ek: Image: derekwaynecarr/memhog Port: Command: memhog -r100 100m Limits: cpu: 100m memory: 200Mi Environment: <noneMounts: <noneVolumes: <noneEvents: FirstSeen LastSeen Count From SubObjectPath Type Reason Message --------- -------- ----- ---- ------------- -------- ------ ------- 3m 3m 1 job-controller Warning FailedCreate Error creating: pods "memhog-ljaqz4ek-" is forbidden: exceeded quota: memoryquota, requested: memory=200Mi, used: memory=400Mi, limited: memory=400Mi Furthermore, pods related to the remaining jobs aren't even created and only the completed pods are shown: $ kubectl get pods -a NAME READY STATUS RESTARTS AGE memhog-0ju2eiuy-bz0zm 0/1 Completed 0 7m memhog-t8qyt5sj-q9msw 0/1 Completed 0 7m Anything else we need to know: I've also created a stackoverflow question about this (), and asked about this in the kubernetes slack, where I was urged to create a bug report.
I will try to reproduce. The creation of a pod will be blocked by quota. In the absence of a pod, the scheduler has nothing to schedule, so it is not a scheduling bug. It's possible the issue is related to back-off in the job controller not attempting to create pods again. The event shows that it had been 3m since the job controller attempted to create pods again for the requisite job which implies some type of back-off. None of the controllers are quota aware, so I would anticipate they should just keep trying at reasonable intervals at this point. If you describe quota after the first 2 pods are completed, do you see quota usage being replenished as expected?
I would like to fix the bug, could you assigin it to me?

if pod create failed, jobcontroller.managejob() need return error, will requeue podcreate task.
-- maybe you have more insight, but it looks to me just reading the code that when a job controller is unable to create a pod due to lack of quota, its not actually an error that forces a re-queue of the job. in addition, i cant actually see when a full resync of the job work queue happens outside of the informer sync interval which is probably really long. see: see:
- feel free to open a pr, please cc me on your solution. i would like to understand from the job folks if there should be a periodic resync interval aside from the need to requeue the job when all pods cannot be spawned (which i assume your pr will handle via exponential backoff)
-- we will want to have a fix for this downstream as well.
Yes, after the first two jobs are complete, shows that 0M out of 400M is in use. I'm also able to create new jobs, up to the memory limit so 2 at a time, that execute as expected. Any jobs over that at a time will stay in the same waiting state.
- Yes, I want handle it via exponential backoff. IMO, MaxOfRateLimiter had implemented a periodic resync, we don't need another one.
I'll make sure to cherry-pick this to openshift ( fyi).


Question: resp := etcdVersionResponse{} var err error versionURL := fmt.Sprintf("%s/%s", endpoint, "version") <del> if tmpVersionURL, err := purell.NormalizeURLString(versionURL, purell.FlagRemoveDuplicateSlashes); err != nil { </del> <ins> if tmpVersionURL, err := normalizeURLString(versionURL); err != nil { </ins> errorList = append(errorList, errors.Wrapf(err, "failed to normalize external etcd version url %s", versionURL)) continue } else {

This is a proactive issue opened to explore the path forward with the module that we have been using as a dependency for in here This module is looking for a new maintainer. Discussions around the path forward, potentially eliminate the module from usage and do this operation in-house, as we use it in only one place for specific functionality. Ref Issue from repo: Slack Thread:
in kubeadm the usage of : Unicode normalization on the URL host according to "Unicode Normalization Form C". Unicode transformation to "fold" characters to canonical width on the host. the host from Unicode to ASCII (IDNA encoding, but 1 and 2 and needed due to a non-conformant x/net/idna at the time?) duplicate "//" from the URL path. the whole URL as per RFC 3986 (looks like PuerkitoBio/urlesc forks golang/net/url). IMO, we should: replace the usage of with this trimmed local version: then add a release note to the PR: we could import , and do the punycode conversion, but i do not understand why it should be kubeadm's responsibility to do that. it feels like IDNA is a host reachability problem that should be solved pre-k8s deployment. i can see this being a breaking change to users having Unicode etcd endpoints in kubeadm, but i'm hoping the same users were already aware to not rely on k8s components to manage this for them.
/sig cluster-lifecycle /area kubeadm /remove-kind bug /kind cleanup
cool, that was helpful. I'll take up on this and create a PR.
/assign
From a Cluster API perspective: It's highly unlikely from the CAPI side that we'll specify IDNA in the kubeadm config. There is a chance on CAPV (vSphere) that a host picks up the DHCP option for the domain name suffix, and that it's using an IDNA. E.g., in my home environment, i have DHCP option 15 set to , and these end up in the etcd pods as parameters, e.g. Is IDNA encoding a consideration for input in etcd's command line, or are unicode args fine?

For the public clouds, it's even more unlikely we'll hit it as the choice of domain name suffix is highly constricted to meet node identity requirements in the CPI. My own take, is that I'm not wed to keeping this around at all, I haven't seen a use case for it.
cc and for any further vsphere comments.
have you seen DHCP with IDNA domains used for external etcd with kubeadm?
unicode args should be fine, also apparently Go's HTTP stack does IDNA encoding: i will update the proposed release note above. but IMO, if one would like a domain to be encoded when passed to all the tools in the k8s stack, maybe it should be done in advance. it feels like IDNA is a host reachability problem that should be solved pre-k8s deployment. or leave it to the Go standard library to do the encoding. i saw comments that it might not be fully compliant yet, but i don't know how up to date these comments are.
/triage accepted i didn't see any PRs for this for 1.22. looks like we can update it in 1.23.
Apologies, I was a little occupied at work. But I have created the PR Please let me know if we can add this as part of 1.22.
no worries. we cannot add it in 1.22 because it's not a critical bug fix and we are in code freeze. i will comment on the PR.
looks like we should tell go-openapi/jsonreference to stop using the library too.
done. Created an issue in their repo as well and offered to create a PR if they are willing to move forward with in-sourcing the logic.
thanks!
just an FYI, the maintainers of the go-openapi have also removed the purell dependency.


Question: // TestErrConnKilledHTTP2 check if HTTP/2 connection is not closed when an HTTP handler panics // The net/http library recovers the panic and sends an HTTP/2 RST_STREAM. func TestErrConnKilledHTTP2(t *testing.T) { <del> readStdErr, cleanUp, err := captureStdErr() if err != nil { t.Fatalf("unable to setup the test, err %v", err) } defer cleanUp() </del> <ins> var buf bytes.Buffer klog.SetOutput(&buf) klog.LogToStderr(false) defer klog.LogToStderr(true) </ins> handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // this error must be ignored by the WithPanicRecovery handler // it is thrown by WithTimeoutForNonLongRunningRequests handler when a response has been already sent to the client and the handler timed out

/sig api-machinery /kind bug /priority important-soon
It seems something is racing writhing the logs ? cc:
/cc /triage accepted
Failed with this: pull-kubernetes-unit
this looks like a test-only race where the test overwriting the global os.Stderr in order to capture error output races with an async attempt to log to it from another test in the same package
which still needs to be fixed
It seems to me that this bug can not be fixed with as it is, namely used without a logger object or even a . The smallest tweak that occurs to me would be to add an option to to add an output destination, and later remove that addition. Thus, a test could add logging to its own buffer for the duration of running the code under test.
This may be similar to . How can I reproduce this issue?


Question: if addr != nil { node.Status.HostIP = node.Name } else { <del> addrs, err := net.LookupIP(node.Name) </del> <ins> addrs, err := lookupIP(node.Name) </ins> if err != nil { glog.Errorf("Can't get ip address of node %s: %v", node.Name, err) } else if len(addrs) == 0 {

It seems like this depends on the network condition as I first noticed this while on a plane, and it happens to me every so often but never on travis. Offenders are TestSyncNodeStatusDeletePods and TestSyncNodeStatusTransitionTime. This may be reproducible by disconnecting network connection but also happens sometimes while connected. is a call to net.LookupIP(), I'll investigate more.
This also occurs on OSX over a VPN (at least, for my VPN it does). Each call to net.LookupIP(name) takes 30s and fails correctly. As a result, the entire unit test times out. That same function call fails fast on Linux.
cc
For other runtime, we can just mock out net, time. I'd really like to know how to do mocking in go. Using interface seems overkill.


Question: return nodeIP } <ins> func detectNumCPU() int { // try get numCPU from /sys firstly due to a known issue (https://github.com/kubernetes/kubernetes/issues/99225) _, numCPU, err := machine.GetTopology(sysfs.NewRealSysFs()) if err != nil || numCPU < 1 { return goruntime.NumCPU() } return numCPU } </ins> func getDetectLocalMode(config *proxyconfigapi.KubeProxyConfiguration) (proxyconfigapi.LocalMode, error) { mode := config.DetectLocalMode switch mode {

A new flag indicating how kube-proxy gets the number of CPUs, either from goruntime or . In kube-proxy, we can use to set node's , the resulting is . We have nodes where pods with none CPU policy and static CPU policy are running together. Pods like kube-proxy are running in none CPU policy by default, and they only gets CPUs left by static CPU policy pods. If static CPU policy pods take away most CPUs cores, kube-proxy can only get one or two cores, then will be set to a very small values. In this case, a lot packet failure can occur. When we set kube-proxy to work in static CPU policy, kube-proxy only gets the number of CPUs selected by the CPU policy, not the number of total CPUs on the node. This may surprise users who are only trying to limit kube-proxy's CPU usages, but resulting in some conntrack fulls faults. Please let me know if this enhancement is reasonable, so I can send a pr later. Thanks!
/sig network
Is it refer to deploy order of kube-proxy? If kube-proxy deploy first in node, will it get just one or two core?
By one or two I mean kube-proxy gets a small number of CPU cores, because most of CPU cores are allocated to static policy pods, sorry for the misunderstanding. The number of conntrack-max depends on the CPU cores when kube-proxy started, even kube-proxy was first deployed, it can get restarted or redeployed by some reasons. Then the conntrack-max will change.
yeah, when kube-proxy deploy firstly, Maybe it can see the whole cpu nums from shared cpu(/var/lib/kubelet/cpumanagestate when kubelet start with static cpu policy). But I think it is a bug for kube-proxy. The cpu info can also get from node resource info. So will it be suitable to add a flag to set cpu num?
Maybe, or we can add a flag for user to choose where kube-proxy should get CPU informations from. There maybe some multi-tenant scenarios where kube-proxy are deployed for per tenant so they would like to limit conntrack-max by CPU cores allocated to each tenant.
I think the simple fix is to replace with something that counts /proc/cpuinfo. I don't know of anyone running multiple proxies, and I don't think it would work well. The intention was "all the CPUs on the node". PRs welcome!
/triage accepted


Question: return allErrs } <del> func validateContainers(containers []core.Container, volumes map[string]core.VolumeSource, fldPath *field.Path) field.ErrorList { </del> <ins> func validateContainers(containers []core.Container, isInitContainers bool, volumes map[string]core.VolumeSource, fldPath *field.Path) field.ErrorList { </ins> allErrs := field.ErrorList{} if len(containers) == 0 {

I found that kubelet will not allocate hostports specified from initContainers. And PodFitsHostPorts in kube-scheduler does not take initContainer into consideration. Also, kube-apiserver think such pod is illegal, but I think it OK. We should make the behavior of spec.HostPorts in initContainer clear
/sig node
Exactly. I think this makes sense. Since initContainers are expected to be completed, we should not expose extra port mappings during creating pod sandbox, which may bring in security risks.
Sounds good to me.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: return errors.Errorf("couldn't retrieve StaticPodSpec for %q", componentName) } <ins> // print all volumes that are mounted for _, v := range spec.Spec.Volumes { klog.V(2).Infof("[control-plane] adding volume %q for component %q", v.Name, componentName) } </ins> // if kustomizeDir is defined, customize the static pod manifest if kustomizeDir != "" { kustomizedSpec, err := staticpodutil.KustomizeStaticPod(&spec, kustomizeDir)

<!-- Please only use this template for submitting enhancement requests --What would you like to be : when using kubeadm init include Why is this needed: the log only show extra-volumes for that control-plane? like this
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig kubeadm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig kubeadm Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/area kubeadm /kind cleanup /sig cluster-lifecycle
what version of kubeadm are you using?
this minor issue is in master too. the fix is in this PR:


Question: return pod, podStats } <ins> func makePodWithLocalStorageCapacityIsolationOpen(name string, priority int32, requests v1.ResourceList, limits v1.ResourceList, memoryWorkingSet string) (*v1.Pod, statsapi.PodStats) { vol := newVolume("local-volume", v1.VolumeSource{ EmptyDir: &v1.EmptyDirVolumeSource{ SizeLimit: resource.NewQuantity(requests.Memory().Value(), resource.BinarySI), }, }) var vols []v1.Volume vols = append(vols, vol) pod := newPod(name, priority, []v1.Container{ newContainer(name, requests, limits), }, vols) var podStats statsapi.PodStats switch name { case "empty-dir": podStats = newPodMemoryStats(pod, *resource.NewQuantity(requests.Memory().Value()*2, resource.BinarySI)) case "container-ephemeral-storage-limit": podStats = newPodMemoryStats(pod, *resource.NewQuantity(limits.StorageEphemeral().Value(), resource.BinarySI)) case "pod-ephemeral-storage-limit": podStats = newPodMemoryStats(pod, *resource.NewQuantity(limits.StorageEphemeral().Value()*2, resource.BinarySI)) default: podStats = newPodMemoryStats(pod, resource.MustParse(memoryWorkingSet)) } return pod, podStats } </ins> func makeMemoryStats(nodeAvailableBytes string, podStats map[*v1.Pod]statsapi.PodStats) *statsapi.Summary { val := resource.MustParse(nodeAvailableBytes) availableBytes := uint64(val.Value())

As I was reading the test code, I noticed that we are missing a pretty large chunk of logic around localStorageCapacityIsolation. This feature was GA in 1.25 so it would be ideal to have unit test coverage around this code. This code monitors whether or not there is a violation in temporary storage and if there is it would evict the pod. One should be able to intiailize the manager with of true and then write some test cases to verify this behavior. /help /sig node /good-first-issue /kind bug
/cc do you have any recollection why there weren't tests for this path?
I can help with this. can you assign me after jingxu replies to you?(for more context here)
that would be great !
you're welcome. do you have any context here that why didn't cover localStorageCapacityIsolation?
I don't recall a specific reason. The feature was implemented a while back.
/assign
Hey Are you still working. If not can I take this up?
hi, yes I'm still working on it
hii I am new to k8s , so i wanna know can I use windows os for it? if yes would you show me how to get nodes...
hi, the answer is yes you can use windows and I think if you need support you better open another issue and @ me there, as this issue is not for discuss how to run k8s on windows
/remove-kind bug /kind feature /triage accepted
I see and I am wondering if we have duplicated efforts here. did you end up testing this path in our above PR?
Hey, is someone working on this issue ? or let me know if someone need help
Happy to help with this. Do you still need help
hey, is this currently still unassigned
has a ready for approval. /remove-good-first-issue /remove-help


Question: <ins> /* Copyright 2021 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package network import ( "context" "fmt" "net" "strconv" "github.com/onsi/ginkgo" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/intstr" clientset "k8s.io/client-go/kubernetes" "k8s.io/kubernetes/test/e2e/framework" e2enode "k8s.io/kubernetes/test/e2e/framework/node" e2epod "k8s.io/kubernetes/test/e2e/framework/pod" imageutils "k8s.io/kubernetes/test/utils/image" ) var _ = SIGDescribe("HostPort", func() { f := framework.NewDefaultFramework("hostport") var ( cs clientset.Interface ns string ) ginkgo.BeforeEach(func() { cs = f.ClientSet ns = f.Namespace.Name }) /* Release: v1.16, v1.20 Testname: Scheduling, HostPort matching and HostIP and Protocol not-matching Description: Pods with the same HostPort value MUST be able to be scheduled to the same node if the HostIP or Protocol is different. This test is marked LinuxOnly since hostNetwork is not supported on Windows. */ framework.ConformanceIt("validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly]", func() { localhost := "127.0.0.1" family := v1.IPv4Protocol if framework.TestContext.ClusterIsIPv6() { localhost = "::1" family = v1.IPv6Protocol } // Get a node where to schedule the pods nodes, err := e2enode.GetBoundedReadySchedulableNodes(cs, 1) framework.ExpectNoError(err) if len(nodes.Items) == 0 { framework.Failf("No nodes available") } nodeName := nodes.Items[0].Name ips := e2enode.GetAddressesByTypeAndFamily(&nodes.Items[0], v1.NodeInternalIP, family) if len(ips) == 0 { framework.Failf("Failed to get NodeIP") } hostIP := ips[0] port := int32(54321) // Create pods with the same HostPort ginkgo.By(fmt.Sprintf("Trying to create a pod(pod1) with hostport %v and hostIP %s and expect scheduled", port, localhost)) createHostPortPodOnNode(f, "pod1", ns, localhost, port, v1.ProtocolTCP, nodeName) ginkgo.By(fmt.Sprintf("Trying to create another pod(pod2) with hostport %v but hostIP %s on the node which pod1 resides and expect scheduled", port, hostIP)) createHostPortPodOnNode(f, "pod2", ns, hostIP, port, v1.ProtocolTCP, nodeName) ginkgo.By(fmt.Sprintf("Trying to create a third pod(pod3) with hostport %v, hostIP %s but use UDP protocol on the node which pod2 resides", port, hostIP)) createHostPortPodOnNode(f, "pod3", ns, hostIP, port, v1.ProtocolUDP, nodeName) // check that the port is being actually exposed to each container // create a pod on the host network in the same node hostExecPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "e2e-host-exec", Namespace: f.Namespace.Name, }, Spec: v1.PodSpec{ HostNetwork: true, NodeName:    nodeName, Containers: []v1.Container{ { Name:  "e2e-host-exec", Image: imageutils.GetE2EImage(imageutils.Agnhost), }, }, }, } f.PodClient().CreateSync(hostExecPod) // use a 5 seconds timeout per connection timeout := 5 // IPv6 doesn't NAT from localhost -> localhost, it doesn't have the route_localnet kernel hack, so we need to specify the source IP cmdPod1 := []string{"/bin/sh", "-c", fmt.Sprintf("curl -g --connect-timeout %v --interface %s http://%s/hostname", timeout, hostIP, net.JoinHostPort(localhost, strconv.Itoa(int(port))))} cmdPod2 := []string{"/bin/sh", "-c", fmt.Sprintf("curl -g --connect-timeout %v http://%s/hostname", timeout, net.JoinHostPort(hostIP, strconv.Itoa(int(port))))} cmdPod3 := []string{"/bin/sh", "-c", fmt.Sprintf("nc -vuz -w %v %s %d", timeout, hostIP, port)} // try 5 times to connect to the exposed ports success := false for i := 0; i < 5; i++ { // check pod1 ginkgo.By(fmt.Sprintf("checking connectivity from pod %s to serverIP: %s, port: %d", hostExecPod.Name, localhost, port)) hostname1, _, err := f.ExecCommandInContainerWithFullOutput(hostExecPod.Name, "e2e-host-exec", cmdPod1...) if err != nil { framework.Logf("Can not connect from %s to pod(pod1) to serverIP: %s, port: %d", hostExecPod.Name, localhost, port) continue } // check pod2 ginkgo.By(fmt.Sprintf("checking connectivity from pod %s to serverIP: %s, port: %d", hostExecPod.Name, hostIP, port)) hostname2, _, err := f.ExecCommandInContainerWithFullOutput(hostExecPod.Name, "e2e-host-exec", cmdPod2...) if err != nil { framework.Logf("Can not connect from %s to pod(pod2) to serverIP: %s, port: %d", hostExecPod.Name, hostIP, port) continue } // the hostname returned has to be different because we are exposing the same port to two different pods if hostname1 == hostname2 { framework.Logf("pods must have different hostname: pod1 has hostname %s, pod2 has hostname %s", hostname1, hostname2) continue } // check pod3 ginkgo.By(fmt.Sprintf("checking connectivity from pod %s to serverIP: %s, port: %d UDP", hostExecPod.Name, hostIP, port)) _, _, err = f.ExecCommandInContainerWithFullOutput(hostExecPod.Name, "e2e-host-exec", cmdPod3...) if err != nil { framework.Logf("Can not connect from %s to pod(pod2) to serverIP: %s, port: %d", hostExecPod.Name, hostIP, port) continue } success = true } if !success { framework.Failf("Failed to connect to exposed host ports") } }) }) // create pod which using hostport on the specified node according to the nodeSelector // it starts an http server on the exposed port func createHostPortPodOnNode(f *framework.Framework, podName, ns, hostIP string, port int32, protocol v1.Protocol, nodeName string) { hostPortPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName, }, Spec: v1.PodSpec{ Containers: []v1.Container{ { Name:  "agnhost", Image: imageutils.GetE2EImage(imageutils.Agnhost), Args:  []string{"netexec", "--http-port=8080", "--udp-port=8080"}, Ports: []v1.ContainerPort{ { HostPort:      port, ContainerPort: 8080, Protocol:      protocol, HostIP:        hostIP, }, }, ReadinessProbe: &v1.Probe{ Handler: v1.Handler{ HTTPGet: &v1.HTTPGetAction{ Path: "/hostname", Port: intstr.IntOrString{ IntVal: int32(8080), }, Scheme: v1.URISchemeHTTP, }, }, }, }, }, NodeName: nodeName, }, } _, err := f.ClientSet.CoreV1().Pods(ns).Create(context.TODO(), hostPortPod, metav1.CreateOptions{}) framework.ExpectNoError(err) err = e2epod.WaitForPodNotPending(f.ClientSet, ns, podName) framework.ExpectNoError(err) } </ins>

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: We want to change ownership of a test from scheduler to network Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Once merges, we will exclude hostPort test on Windows. This is because of how the hostPort test is structured where we are relying on hostNetwork pod to test the communication between hostPort pods. This will not work on Windows as hostNetwork is not supported. I think it is better to have this test in sig-network and test the communication as scheduler should be responsible only for assigning a pod to node. This also ensures that hostPort on Windows is exercised properly. Anything else we need to know: More context
/sig scheduling /sig network cc
/area conformance /area test /sig windows since windows
/triage accepted


Question: klog.Errorf("Failed to get VM object for node: %q. err: +%v", convertToString(nodeName), err) return "", err } <del> isActive, err := vm.IsActive(ctx) </del> <ins> exists, err := vm.Exists(ctx) </ins> if err != nil { <del> klog.Errorf("Failed to check whether node %q is active. err: %+v.", convertToString(nodeName), err) </del> <ins> klog.Errorf("Failed to check whether node %q still exists. err: %+v.", convertToString(nodeName), err) </ins> return "", err } <del> if isActive { </del> <ins> if exists { </ins> return vs.vmUUID, nil } <del> klog.Warningf("The VM: %s is not in %s state", convertToString(nodeName), vclib.ActivePowerState) </del> <ins> klog.Warningf("The VM: %s doesn't exist", convertToString(nodeName)) </ins> return "", cloudprovider.InstanceNotFound }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We have k8s cluster with version . And I noticed that if I deleted worker node VM from IAAS, then node stuck in state, I created statefulset before and the pod scheduled on that node was stuck in state, and no new pod was created to replace that. Node should be deleted if no instance is found Related log When I check the log of I could see following log keeps popping up. Looks like node_controller noticed that the VM is gone but it doesn't delete the node. Related code here will return if no instance is found. But it should check if it's VM is deleted not shutdown, then return And it should cherry pick this fix avoiding deleting shutdown node What you expected to happen: After VM is deleted, the node should be deleted, pods assigned to the node should be rescheduled to other available nodes How to reproduce it (as minimally and precisely as possible): k8s cluster. In our case we use PKS. workloads on node worker node VM from IAAS Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: vSphere Cloud Provider
/sig cloud-provider /sig vmware
Same issue as this one The issue and is cluster API repo. We should fix it inside node_controller in k/k
Hi this issue was found when testing windows cluster deployed using PKS. We do have a workaround by manually deleting node using kubectl. Still we wish the fix could go in ASAP. Could you ask someone from upstream to to review the PR that fix this issue? Thank you
, we will try to push this through for v1.20. 1.19 is approaching code thaw, so we can't push this through
Thanks Michael. I'll let my team know.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: }, }, }) <del> if err == nil || !strings.Contains(err.Error(), "namespace") { t.Fatalf("expected error that namespace was missing from context") </del> <ins> // Accept "namespace" or "Namespace". if err == nil || !strings.Contains(err.Error(), "amespace") { t.Fatalf("expected error that namespace was missing from context, got: %v", err) </ins> } }

Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: check Anything else we need to know: /cc /milestone v1.17 /priority critical-urgent
/sig cluster-lifecycle These SIGs are my best guesses for this issue. Please comment if I am incorrect about one. I am a bot run by vllry.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopening to obvserve the test after the fix was merged.
we updated our jobs after the creation of the new branch. there is now: 1-17-on-1-16 needs more runs, to verify if the fix is valid.
this should be resolved yet still flaky due to sig-apps/StatefulSets and some other conditions.
/close
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: t.Errorf("Expected IP address: %s, saw %s", nodeIP, *routeInfo.NextHopIPAddress) } <ins> // test create again without real creation, clean fakeRoute calls fakeRoutes.Calls = []string{} routeInfo.Name = &routeName route.Name = routeName expectedTable.RouteTablePropertiesFormat = &network.RouteTablePropertiesFormat{ Routes: &[]network.Route{routeInfo}, } cloud.rtCache.Set(cloud.RouteTableName, &expectedTable) err = cloud.CreateRoute(context.TODO(), "cluster", "unused", &route) if err != nil { t.Errorf("unexpected error creating route: %v", err) t.FailNow() } if len(fakeRoutes.Calls) != 0 { t.Errorf("unexpected route calls create if not exists, exists: %v", fakeRoutes.Calls) } </ins> // test create route for unmanaged nodes. nodeName := "node1" nodeCIDR := "4.3.2.1/24"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Kubernetes shouldn't put route table when it's already been configured. Azure cloud provider should check whether the route has already been created before invoking . What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /sig cloud-provider /priority important-soon
Why this is a bug? A PUT operation should be idempotent and multiple calls should get the same result?
Right. But cloud provider shouldn't make unnecessary API calls.
To check whether it is configured, we need a GET call, so we're going to replace PUT call with GET call in this case?
it's not a replace, we need to LIST routes and check whether it has already been there, then PUT if it's not.
I got your point. But if we directly send a PUT request, seems route RP will also do this comparation? Is there any issue making us have to do the comparation by ourselves instead of sending PUT request to route RP?
It would help to reduce the chances of ARM PUT throttling.
I'd like to work on this
/assign


Question: go_library( name = "go_default_library", <del> srcs = ["label.go"], </del> <ins> srcs = [ "label.go", "label_selinux.go", ], </ins> importmap = "k8s.io/kubernetes/vendor/github.com/opencontainers/selinux/go-selinux/label", importpath = "github.com/opencontainers/selinux/go-selinux/label", visibility = ["//visibility:public"], <ins> deps = select({ "@io_bazel_rules_go//go/platform:android": [ "//vendor/github.com/opencontainers/selinux/go-selinux:go_default_library", ], "@io_bazel_rules_go//go/platform:linux": [ "//vendor/github.com/opencontainers/selinux/go-selinux:go_default_library", ], "//conditions:default": [], }), </ins> ) filegroup(

What happened: After upgrading from kubelet 1.15 -1.16, volumes are no longer relabeled as containerfilet for selinux, causing most pods to fail. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Run a kubernetes cluster on 1.15.[0-3]. Create a pod with a secret, configmap, or with provisioned storage (ebs-csi-driver). Find that file on disk of the node, and observe it is labeled with the file type containerfilet and the pod is able to read it. Upgrade the kubelet to 1.16 Delete the pod, and observe it fail on launch. Find the file on disk and see it labeled tmpfst or unlabeledt for configmap/secret and provisioned volumes, respectively. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 / 1.16 Cloud provider or hardware configuration: AWS OS (e.g: ): RHEL 7.7 Kernel (e.g. ): 3.10.0-1062.1.1.el7.x86_64 Install tools: kubeadm Network plugin and version (if this is a network-related bug): N/A Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is selinux enabled for docker? Can you paster output of : also
Also please paste YAML of one or more pods which are affected by this. Thank you!
Security Options - Perhaps of note here, we do not change the installed by docker, and confirms we don't make changes to that file. Docker version: Client and server blocks match. Pod yaml incoming
To clarify, do you want a runnable "minimum reproducible" , or is it more informational? If the latter, I have a pod that breaks and I'm working on the yaml for that, removing sensitive information, etc. If the former, I'll need to spend a few minutes getting a minimal reproduction with busybox or somesuch.
I need full pod's yaml that is not working. i.e - generated full YAML.

Hi, I'm sorry I haven't had a chance to follow up with this ticket. I will get some pod yaml attached early next week. Thank you for following up.
I think I'm seeing the same thing, to reproduce, use the following resource YAML: On k8s v1.15.5 (hyperkube) On k8s 1.16.2 (hyperkube) Are you in Enforcing mode () ?
Hey thanks much for following on with example YAML. We've been totally swamped with work so I haven't had time to address this issue. To answer your question: we were seeing all pods with storage (configmaps, secrets, persistent volumes) failing in Enforcing mode, and disabling Enforcement () allowed the same pods to successfully run, with no problem. There was not as much output in audit logs as we would have expected, so constructing custom policies to allow access was a bit of shooting in the dark. We were able to get configmaps and secrets to work passably with a custom policy allowing limited access to tmpfs labeled items, but we can't get away with allowing access to unlabeledt in our compliance environment. Thanks again for following up with more info and linking up this ticket, and let me know if I can provide any more helpful information. We've rolled back to 15.3 for the time being, so I'll be constructing knowledge somewhat from memory, but the upgrade was run recently, and I took reasonably detailed notes, so my info is be largely accurate.
cc
Can someone post pod's YAML after pod was created. I am looking for some generated fields in the Pod.
I finally had the chance today to dig a bit deeper why SELinux is broken in 1.16+ In the linked PR you can see the fix. See the details in there. If you want to give that a try, I have build a binaries here: A workaround I have used in the meantime was: My reproduction of the bug can be found there: If you run the packer build the pods fail because of the SELinux problem
nice work! looks like the patch we need is this one right?
if so, can you please propose a PR against master of kubernetes/kubernetes?
Reopening to track adding a test.
On second thought, spawned a separate issue to track adding test coverage. /close
Closing this issue.

<detailsIn response to : On second thought, spawned a separate issue to track adding test coverage. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
we have just updated our cluster to 1.16.5, and even 1.16.6-beta.0, but the secret volume is still labeled as tmpfst:
I see the same result on the test above on v1.17.2
I can confirm the release binaries are not build correctly I have opened this issue in the release repo: /reopen Workaround is building the binaries yourself using bazel and the release tag
Reopened this issue. <detailsIn response to : I can confirm the release binaries are not build correctly I have opened this issue in the release repo: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can someone confirm that 1.16.7 / 1.17.3 binaries are built with the selinux tags in make targets?
No they haven't as haven't been merged. Feel free to ask on those PRs again, I have not been successful, convincing that this should be merged with at least some priority
I am able to run this repro and get the desired outcome on 1.17.4. However, I am trying to attach Cinder volume whose context doesn't automatically get relabeled as containerfilet when I attempt to attach volumes with both in-tree and Cinder CSI plugin. I have filed this bug in the cloud-provider-openstack repo but not sure if this is a Kubernetes problem or a cinder-csi-plugin problem:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /assign are looking at adding selinux to CI using kops
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: options.captureProfile(outputProfile) } <ins> // Cache metric streams for prefilter and filter plugins. for i, pl := range f.preFilterPlugins { f.preFilterPlugins[i] = &instrumentedPreFilterPlugin{ PreFilterPlugin: f.preFilterPlugins[i], metric:          metrics.PluginEvaluationTotal.WithLabelValues(pl.Name(), metrics.PreFilter, f.profileName), } } for i, pl := range f.filterPlugins { f.filterPlugins[i] = &instrumentedFilterPlugin{ FilterPlugin: f.filterPlugins[i], metric:       metrics.PluginEvaluationTotal.WithLabelValues(pl.Name(), metrics.Filter, f.profileName), } } </ins> return f, nil }

We observed degradation in scheduling throughput, putting it under 100 pods/s. Respective pprof of kube-scheduler points it to ! /sig scheduling /kind regression 100 pods/node n/a No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig scalability
Thanks for noticing it /assign
just to confirm this bug is present only in 1.27 ( introduced with pluginevaluationtotal metric) , isn't it ?
Exactly, 1.27+ (so the current master as well) to be specific.


Question: // This call is idempotent, and must not return an error if the sandbox has // already been removed. rpc RemovePodSandbox(RemovePodSandboxRequest) returns (RemovePodSandboxResponse) {} <del> // PodSandboxStatus returns the status of the PodSandbox. </del> <ins> // PodSandboxStatus returns the status of the PodSandbox. If the PodSandbox is not // present, returns an error. </ins> rpc PodSandboxStatus(PodSandboxStatusRequest) returns (PodSandboxStatusResponse) {} // ListPodSandbox returns a list of PodSandboxes. rpc ListPodSandbox(ListPodSandboxRequest) returns (ListPodSandboxResponse) {}

The behavior when requested id doesn't exist is not defined clearly in CRI. Currently, we define that should return when requested image doesn't exist (), and kubelet is relying on this behavior now (). However, we haven't clearly defined the behavior of and . Currently, they return error when requested sandbox/container doesn't exist, and kubelet is also relying on this behavior. Not super urgent, just track the issue here. /cc
/assign
Added for clarifying the behavior.


Question: clusters = f.GetRegisteredClusters() ns = f.FederationNamespace.Name // create backend service <del> service = createLBServiceOrFail(f.FederationClientset, ns, FederatedIngressServiceName) </del> <ins> service = createLBServiceOrFail(f.FederationClientset, ns, FederatedIngressServiceName, clusters) </ins> // create the TLS secret secret = createTLSSecretOrFail(f.FederationClientset, ns, FederatedIngressTLSSecretName) // wait for services objects sync

In order to support running tests in parallel, we randomized NodePort selection. There appears to be some evidence that this is not sufficient and is causing e2e test flakes: https://k8s-
Hi, As a need contributor, I'm interesting of solving this issue. I already check the current code, and propose to retry the service creation until success or all possible ports (between FederatedSvcNodePortFirst and FederatedSvcNodePortLast) have been tried. If this approach is fine for you, I can submit a PR soon.
How do you plan to check if the NodePorts are available? In our presubmit test environment, tests run across many instances of federation control planes in parallel, but share the same underlying clusters. So this approach might be viable depending on how and where you plan to check for these used NodePorts.
my approach for getting an available port is a "Dummy" approach: create a list of possible available port, get one of them randomly, then try to create the Service. If Service creation failed. try to use another possible available port, until service creation success or all possible available ports have been tested.
Sounds reasonable to me. Go for it. A couple of caveats: need to create a real list, you could just use the range boundaries. careful about service creation failure. You need to check if the service shard creation failed in the underlying clusters and rollback all those shards before retrying.
thanks for the caveats, I will do soon a PR.
Hi, what is the state of this issue? Is working in this issues with the others issues related to e2e tests?
has a PR to fix this issue -


Question: visible at runtime in the container. release: v1.9 file: test/e2e/common/node/downwardapi.go <del> - testname: Pod events, verify event from Scheduler and Kubelet codename: '[sig-node] Events should be sent by kubelets and the scheduler about pods scheduling and running  [Conformance]' description: Create a Pod, make sure that the Pod can be queried. Create a event selector for the kind=Pod and the source is the Scheduler. List of the events MUST be at least one. Create a event selector for kind=Pod and the source is the Kubelet. List of the events MUST be at least one. Both Scheduler and Kubelet MUST send events when scheduling and running a Pod. release: v1.9 file: test/e2e/node/events.go </del> - testname: init-container-starts-app-restartalways-pod codename: '[sig-node] InitContainer [NodeConformance] should invoke init containers on a RestartAlways pod [Conformance]'

Related: Noticed this as part of Opened in response to: According to our , the e2e test is ineligible for Conformance. Specifically, the test in question is looking for guaranteed delivery of Events from specific sources (kubelet and default scheduler). This cannot be reasonably expected of all Kubernetes clusters, and thus the test should be demoted from Conformance. For historical context: It looks like this has been part of Conformance sometime since sometime prior to 2015-10-17 (we introduced the [Conformance] tag in 1.2, ref: We overhauled Conformance testing policy (including exclusion of Events) on 2018-10-08 (ref: ) If I wanted to be a language lawyer about this, I could say the policy covers ineligibility for promotion but said nothing about the existing corpus of tests. But I think this was just an honest oversight when we reviewed the existing Conformance tests. Due to the very nature of what this test is verifying, I don't think it's possible to rewrite to avoid Events. Thus, I think it should be demoted from Conformance. SIG Node is free to continue to treat this as release-blocking, but it's not something we can reasonably expect of any Kubernetes cluster /kind cleanup /priority important-soon /milestone v1.23 /sig architecture /area conformance because conformance /sig node because it's a sig-node test
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Opened to address FYI Since y'all were commenting on
+1


Question: os:                 kubecontainer.RealOS{}, podSandboxImage:    podSandboxImage, streamingRuntime: &streamingRuntime{ <del> client: client, // Only the native exec handling is supported for now. // TODO(#35747) - Either deprecate nsenter exec handling, or add support for it here. execHandler: &dockertools.NativeExecHandler{}, </del> <ins> client:      client, execHandler: execHandler, </ins> }, containerManager: cm.NewContainerManager(cgroupsName, client), }

Either deprecate the nsenter exec handler, or add nsenter handling to the dockershim (after goes in). /cc
I vote for deprecating the nsenter exec handler for docker. has been the default exec handler since kubernetes pre-1.0. Is there any reason (or use case) for keeping the pluggable exec handler? Another question is that whether we can drop exec in one release, or we need to provide support in CRI for at lease one more release. /cc
+1 for docker exec.
I suggest deprecating docker nsenter exec in a long term, but to achieve that, we need to follow the deprecation policy. The first thing we need to address is marking this flag (--docker-exec-handler, default: native) being deprecated:
Also there are several issues raised before which are related to the issue of docker exec in various versions. The engineer even suggested to switch the default mechanism is docker nsenter exec. One of the examples is
Now let's narrow down to the CRI scope, and below is my 2 cents: 1) Can we mark the flag deprecated in 1.6? 2) Drop the support in CRI in 1.6? Inform the kubernetes users with this decision along CRI announcement being scheduled sometimes in February? 3) Drop the support completely from Kubernetes unless there are real use cases coming up.
We'll have to tell the users that they need to disable CRI in order to use the nsenter exec plugin. could you give a quick estimate about the amount of work required to add support for nsenter exec in dockershim? Thanks.
Talked to offline. I'll Add support nsenter as exec in dockershim Mark the flag as deprecated.


Question: "addIf": "%s", "isGateway": true, "ipMasq": false, <del> "hairpin": "%t", </del> <ins> "hairpinMode": %t, </ins> "ipam": { "type": "host-local", "subnet": "%s",

From another issue, said: DockerManager.SyncPod() calls the network Plugin and then sets hairpin mode unilaterally. This is wrong. We should either pass the bool down to here or else we should get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this. This should be a relatively easy one for people who want to contribute but don't know where to start.
I volunteer myself to do this. Actually I have done something about kubernetes network plugin so guess I can do this. I will start to do this when I am on the office tomorrow.
Awesome. I'd advise starting with a sketch so we can all make sure it is a good approach, before jumping in with both feet. , TonyAdo wrote:
ok, I will start with a sketch as soon as possible.
I thought about this for a while: if we pass the bool down to here, of course we can check this flag, if it is set, we should set veth hairpin mode, but problem is that if user set this flag, but still use a custom network plugin just like you mentioned the ovs plugin, I am afraid we will still enable hairpin mode in all interfaces on this host, since we cannot find a peer interface. Is that correct?
Yeah, I think the better fix is to use the existence of a network plugin as an indicator. It sort of makes --configure-cbr0 redundant , TonyAdo wrote:
ok, I will try to get rid of the default no-op plugin and use the non-existence of a plugin as a flag to control this.
I decided to use the existence of a network plugin as an indicator, also I will get ride of the no-op network plugin, I am not going to add new flag to kubelet structure, if the user don't specify network plugin name, the networkPlugin within kubelet will be which just means non-existence of a plugin, and we have to check the existence in the docker manager to avoid nil pointer exception also for setting, I think if a custom network plugin exists, kubelet should not do this. How about this? Should I open a PR?
fyi, saw a hairpin pr today that might be related. ymmv.
what do you think about this?
What you described sounds right to me.

I think the only way to see if there is some detail I am missing is to try it and see if anything pops up that would make this not work. , TonyAdo wrote:
I need to check which part is responsible of what, but what if the hairpin package was just a utility package that should be called by the plugin if it needs to set this up? As long as it is single and simple function call, doing so would avoid adding extra logic here and fix the issue. This is valid only if the aim is to have a "linux bridge plugin", an "OVS plugin", etc.
actually kubelet will call after get called, I think to make the network plugin can call hairpin package is not that easy, of course, I agree that network plugin should setup hairpin mode if it really needs, kubelet should not set hairpin mode unilaterally. Besides, kubelet is not going to provide an "linux bridge plugin" or "OVS plugin".
then the validity conditions of my comment are not met, so I see any simpler way. Thanks for checking! :)
still what do you think about disable hairpin completely and leave this to network plugin?
or if there is a custom network plugin, we don't setup hairpin mode, only do this in default.
Completely? It will break my use case (and anyone using Apache Kafka).
Only by default is what I understood from the previous comments, and LGTM.
Really sorry about that, but I really want to know are you using a kubelet network plugin in your use case? If no, I think we will still setup hairpin, as in default we will do this.
I don't use any plugin, so it is ok if hairpin is set up by default.
ok, I think this will not break your use case.
It's not really the network plugin that needs hairpinmode though, it's the pure-iptables-based kube-proxy code. It's possible the proxying code could change again in the future in such a way that hairpinmode wouldn't be needed any more. So it seems wrong to put that code into the plugin. makes it so that we set hairpinmode on the veth if it's attached to a bridge, but just silently do nothing if it's not.

(Yes, we currently set hairpinmode even if you're using the userspace proxier, but AFAIK it's not needed in that case.)
IMHO... The proxy has the responsibility of service objects, while the kubelet is responsible of pod objects, including their networking. Also, the responsibility of hairpin must be at the same place as the code that creates the network interface, otherwise you have code managing a single object (the pod interface) spread over multiple programs, which will eventually lead to complex problems and a system harder to understand even for the end-user. Thus, it doesn't seems wrong to me that the kubelet does whatever is required to make the pod work in the environment of the cluster. While its true that the hairpin requirement is only for the proxy in iptables mode, a switch to this mode can happen at any time in the life of the cluster. And when it happens, its probably because the userland proxy becomes unresponsive, which means in a situation where the administrator is in a hurry to find a solution. Currently, you just change the proxy mode and restart the proxies. There's little chance the administrator will think about hairpin or restarting the kubelet in this case, so the kubelet should be reacting to this; which means the proxy and the kubelet of a given host should be able to communicate (AFAIK, this doesn't exist now). And all this is currently only to avoid setting up hairpin on the pod's port of standard Linux bridge.
Feel sure I'm missing something but... The CNI bridge plugin now has a flag, so kubenet can use that. Having done that, we can set false when the plugin name is non-blank.
Is this fixed now?
I apologise for leaving a somewhat vague answer, but I have been in that code recently. I don't think it's fixed.
, Bryan Boreham wrote:
Thanks for the PR. CNI Bridge plugin is expecting . Not . Hence the bridge plugin did not recognize it.
EDIT 2016-12-09: The chain below was broken by , so the issue can be closed again (remain closed) I suggest this should be re-opened. The chain of calls is: sets to , e.g. when using CNI calls something goes wrong and it sets the mode on everything. An example of "something goes wrong":


Question: import ( "context" <del> "errors" </del> "fmt" "os" "path/filepath"

Failing: ` Failing since . Last passing: First failing:
The diff () shows that went in which looks related
/sig node
/kind regression /kind bug /triage accepted


Question: fi } <ins> function delete-cloud-nat-router() { if [[ ${GCE_PRIVATE_CLUSTER:-} == "true" ]]; then if [[ -n $(gcloud compute routers describe --project "${NETWORK_PROJECT}" --region "${REGION}" "${NETWORK}-nat-router" --format='value(name)' 2>/dev/null || true) ]]; then echo "Deleting Cloud NAT router..." gcloud compute routers delete --project "${NETWORK_PROJECT}" --region "${REGION}" --quiet "${NETWORK}-nat-router" fi fi } </ins> function delete-subnetworks() { # If running in custom mode network we need to delete subnets manually. mode="$(check-network-mode)"

Public IPs are expensive resources. Given that we're running scalability tests on 5k-node clusters, we are unnecessarily consuming 5k IPs which is a lot. We should switch our tests so that nodes have only private IP addresses. We are going to do that by configuring NAT so that nodes can talk to outside world (e.g. to download image from dockerhub). But that also requires changing some correctness tests which assume public IPs.
To summarize recent work: setting up private gce clusters with nat, feature is gated via the env test and dashboard with correctness tests on gce private cluster - https://k8s- 27 tests fail on fixing the tests - should fix majority of them
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
- is there anything missing to be done here?
No, we can close this one. It was done long time ago. Thanks for catching /close
Closing this issue. <detailsIn response to : No, we can close this one. It was done long time ago. Thanks for catching Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: return err } <ins> // avoid repeated calculation for the same generation c.lastSeenGenerationLock.Lock() lastSeen, seenBefore := c.lastSeenGeneration[inCustomResourceDefinition.Name] c.lastSeenGenerationLock.Unlock() if seenBefore && inCustomResourceDefinition.Generation <= lastSeen { return nil } </ins> // check old condition cond := calculateCondition(inCustomResourceDefinition) old := apiextensions.FindCRDCondition(inCustomResourceDefinition, apiextensions.NonStructuralSchema)

/sig api-machinery /kind bug /assign sttts because the controller that maintains the condition runs multiple instances in HA apiserver clusters, the controller must not duel in cases where the validation messages change slightly between releases


Question: objects. It performs two major functions: cluster-wide node synchronization and single node life-cycle management. <del> Node controller has a sync loop that creates/deletes Nodes from Kubernetes </del> <ins> Node controller has a sync loop that deletes Nodes from Kubernetes </ins> based on all matching VM instances listed from the cloud provider. The sync period can be controlled via flag `--node-sync-period`. If a new VM instance gets created, Node Controller creates a representation for it. If an existing

The expression right now is: "Node controller has a sync loop that creates/deletes Nodes from Kubernetes based on all matching VM instances listed from the cloud provider." But actually node controller does not create nodes from cloudprovider. Please refer to This issue was logged for the question with answer given: So the document expression should remove "creates" from the sentence.
Can we close this now that was merged?
Sure, close this issue. Thanks.


Question: } } <ins> // TestAddPodAlwaysUpdatePodInfoInNodeInfo tests that AddPod method always updates PodInfo in NodeInfo, // even when the Pod is assumed one. func TestAddPodAlwaysUpdatesPodInfoInNodeInfo(t *testing.T) { ttl := 10 * time.Second now := time.Now() p1 := makeBasePod(t, "node1", "test-1", "100m", "500", "", []v1.ContainerPort{{HostPort: 80}}) p2 := p1.DeepCopy() p2.Status.Conditions = append(p1.Status.Conditions, v1.PodCondition{ Type:   v1.PodScheduled, Status: v1.ConditionTrue, }) tests := []struct { podsToAssume         []*v1.Pod podsToAddAfterAssume []*v1.Pod nodeInfo             map[string]*framework.NodeInfo }{ { podsToAssume:         []*v1.Pod{p1}, podsToAddAfterAssume: []*v1.Pod{p2}, nodeInfo: map[string]*framework.NodeInfo{ "node1": newNodeInfo( &framework.Resource{ MilliCPU: 100, Memory:   500, }, &framework.Resource{ MilliCPU: 100, Memory:   500, }, []*v1.Pod{p2}, newHostPortInfoBuilder().add("TCP", "0.0.0.0", 80).build(), make(map[string]*framework.ImageStateSummary), ), }, }, } for i, tt := range tests { t.Run(fmt.Sprintf("case_%d", i), func(t *testing.T) { cache := newCache(ttl, time.Second, nil) for _, podToAssume := range tt.podsToAssume { if err := assumeAndFinishBinding(cache, podToAssume, now); err != nil { t.Fatalf("assumePod failed: %v", err) } } for _, podToAdd := range tt.podsToAddAfterAssume { if err := cache.AddPod(podToAdd); err != nil { t.Fatalf("AddPod failed: %v", err) } } for nodeName, expected := range tt.nodeInfo { n := cache.nodes[nodeName] if err := deepEqualWithoutGeneration(n, expected); err != nil { t.Errorf("node %q: %v", nodeName, err) } } }) } } </ins> // TestAddPodWillReplaceAssumed tests that a pod being Add()ed will replace any assumed pod. func TestAddPodWillReplaceAssumed(t *testing.T) { now := time.Now()

Some Pods in Kubernetes scheduler's internal cache don't have condition in their status. We store scheduled pods information into cache as assumedPod before binding Pod. And, just after binding Pod, we receive event for scheduled Pods from eventhandler and will be used to add Pod to cache. But, as described above, we've store scheduled pods information. So, basically, scheduler doesn't update pod in cache here: But, event contains changes not only of but also of . So, we should update cache to keep up-to-date when we receive event for scheduled Pods from eventhandler. Otherwise, won't updated until new changes for that Pod are made. Kubernetes scheduler's internal cache always keeps Pod's up-to-date. some Node. a Pod. is scheduled by scheduler. in Pods in Kubernetes scheduler's internal cache (by adding debug log or cache) I think kubelet will update Pods status after they run containers, so you need to do (4) before kubelet does so. (Or doesn't use kubelet like e2e tests) /assign Let me double-check this. (The above explanation includes my expectations, so may not be entirely correct.) v1.23.4 But, I guess this bug exists on latest master as well. <details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale
Does this lead to unschedulable pods or is this just for debugging purposes?
We uses condition in a custom plugin. Mostly No in the real cluster. Only for a very short time (from the scheduler schedules a Pod to kubelet next updates a pod) our plugin does not behave as expected and may make some pods unschedulable. But, when it comes to e2e testing, this bug can be troublesome. In our e2e testing, we don't run kubelet, and no one updates Pods after the scheduler scheduled them Then, all scheduled Pods in the scheduler cache have non-updated condition and it results in unexpected results from our plugin.


Question: framework.ConformanceIt("Replace and Patch tests", func() { testRSLifeCycle(f) }) <ins> ginkgo.It("should list and delete a collection of ReplicaSets", func() { listRSDeleteCollection(f) }) </ins> }) // A basic test to check the deployment of an image using a ReplicaSet. The

[X] test approval issue : [x] test pr : [x] two weeks soak end date : 2 June 2021 [x] test promotion pr : kubernetes/kubernetes According to this APIsnoop query, there are still some remaining ReplicaSet endpoints which are untested. Create a ReplicaSet List all Replicasets DeleteCollection of ReplicaSets Varify the ReplicaSets have been deleted Using an existing conformance test as a template for a which validates that two new endpoints are hit. example endpoint | useragent --------------------------------------------|
createAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets deleteAppsV1CollectionNamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets listAppsV1ReplicaSetForAllNamespaces | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets readAppsV1NamespacedReplicaSet | (linux/amd64) kubernetes/$Format -- [sig-apps] ReplicaSet should list and delete a collection of ReplicaSets (4 rows) If a test with these calls gets merged, test coverage will go up by 2 points This test is also created with the goal of conformance promotion. /sig apps /sig testing /sig architecture /area conformance
/triage accepted For Release 1.22
/assign
Dear When I run It only returns not listAppsV1ReplicaSetForAllNamespaces. What does this mean? What should I do? Regarding What should I see after implementing the tests? All 4 rows? Right now I see 0 rows when running it (without editing code) Lastly, could you tell me which parts need to be addressed in the ? Right now, it looks like the template does exactly what you want to write for this test.
This issue is created to show a planned PR to the conformance meeting. It is just a proof of concept before creating the conformance PR. What we do: I create a local cluster and create a test. In the cluster, I have running to see what user agents are hit by the test. Before I run the test I delete all cluster logs.

Then I run the test. This is my test: Once the test pass I run the APISnoop query to see which user-agents are hit. I in this case 4 API's are hit, but I am only interested in and as they are the ones yet.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: usagePercent := 100 - int(available*100/capacity) if usagePercent >= im.policy.HighThresholdPercent { amountToFree := capacity*int64(100-im.policy.LowThresholdPercent)/100 - available <del> glog.Infof("[imageGCManager]: Disk usage on image filesystem is at %d%% which is over the high threshold (%d%%). Trying to free %d bytes", usagePercent, im.policy.HighThresholdPercent, amountToFree) </del> <ins> glog.Infof("[imageGCManager]: Disk usage on image filesystem is at %d%% which is over the high threshold (%d%%). Trying to free %d bytes down to the low threshold (%d%%).", usagePercent, im.policy.HighThresholdPercent, amountToFree, im.policy.LowThresholdPercent) </ins> freed, err := im.freeSpace(amountToFree, time.Now()) if err != nil { return err

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig node What happened: following is kubelet logs It's confusing that why kubelet still try to free bytes when usage(85%) equals to high threshold(85%) With the code, I understood that kubelet tried to keep freeing down to the LowThresholdPercen(80%) not the HighThresholdPercent What you expected to happen: How to reproduce it (as minimally and precisely as possible): Reduce the imagefs capacity to trigger image gc Environment: latest Kubernetes
/assign


Question: return nil } <ins> var _translationsOwners = []byte(`reviewers: - brendandburns approvers: - brendandburns `) func translationsOwnersBytes() ([]byte, error) { return _translationsOwners, nil } func translationsOwners() (*asset, error) { bytes, err := translationsOwnersBytes() if err != nil { return nil, err } info := bindataFileInfo{name: "translations/OWNERS", size: 0, mode: os.FileMode(0), modTime: time.Unix(0, 0)} a := &asset{bytes: bytes, info: info} return a, nil } </ins> var _translationsExtractPy = []byte(`#!/usr/bin/env python # Copyright 2017 The Kubernetes Authors.

/assign /cc Please ensure that every sub directory of / contains an owners file. The following sub directories do not define them: The result of this is that any PR which changes anything in api, Godeps, staging, vendor, etc will always get assigned to one of you for both the review and approval.
I recall someone also mentioning that certain files or dirs may not need approvers files. If that would help, we can add that functionality (i've already started the work)
staging and api should be easy to find owners for. may have ideas for translations. The big one is Godeps, third_party, vendor. Those are just grunt work. And figuring out criteria for approving is hard as they are a lot of code and it is unclear that things are tested well or automatically. (I'm thinking things like libraries for accessing cloud APIs). Having a dedicated team/process there might help.
I'll open some proposals for staging and API.
Godep in particular needs to be watched for things like licenses, which I have mostly been doing.
Thanks Clayton for the staging and api additions! How about if we pick two of you at random and shove them in as owners for each of the remaining directories?
What's left: translations: the kubectl specific translations is being handled by sig-cli, but the rest of the files like README and test have been handled and approved mostly by thirdparty: this is a tricky mix of stuff like forked code.. mostly handled by the likes of or for approvals. Godep/vendor: These could probably be solved by having a process to follow, and then delegating to a team in OWNERSALIASES for both folders.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Still relevant. /remove-lifecycle stale


Question: // numFeasibleNodesToFind returns the number of feasible nodes that once found, the scheduler stops // its search for more feasible nodes. <del> func (g *genericScheduler) numFeasibleNodesToFind(numAllNodes int32) int32 { if numAllNodes < minFeasibleNodesToFind || g.percentageOfNodesToScore <= 0 || g.percentageOfNodesToScore >= 100 { </del> <ins> func (g *genericScheduler) numFeasibleNodesToFind(numAllNodes int32) (numNodes int32) { if numAllNodes < minFeasibleNodesToFind || g.percentageOfNodesToScore >= 100 { </ins> return numAllNodes } <del> numNodes := numAllNodes * g.percentageOfNodesToScore / 100 </del> <ins> adaptivePercentage := g.percentageOfNodesToScore if adaptivePercentage <= 0 { adaptivePercentage = schedulerapi.DefaultPercentageOfNodesToScore - numAllNodes/125 if adaptivePercentage < minFeasibleNodesPercentageToFind { adaptivePercentage = minFeasibleNodesPercentageToFind } } numNodes = numAllNodes * adaptivePercentage / 100 </ins> if numNodes < minFeasibleNodesToFind { return minFeasibleNodesToFind } return numNodes }

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Kubernetes scheduler has that specifies what percentage of the cluster nodes should be scored in each scheduling cycle. The scheduler stops scanning more nodes once that many nodes are found feasible for running the pod under consideration. The default value of this config option is 50 (for 50%). We should let the scheduler determine this value based on the size of the cluster when this config option is not provided. I'd propose this formula for finding the percentage: . This is a linear formula that gives us 50 when the cluster size is 100 nodes and 10 for a 5000 node cluster. The scheduler already has a lower bound of 50 nodes per scheduling cycle. As a part of this change, we should also add a 5% lower bound for the formula (If the result of the formula is below 5, we should return 5). Why is this needed: To improve scheduler performance in large clusters. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature /sig scheduling
/assign /remove-help
/assign wgliang
Thanks