Question: assert.Len(t, realServers1, 1, "Expected 1 real servers") assert.Equal(t, realServers1[0].String(), "10.0.1.5:80") <del> // externalIP should have 0 endpoints since the feature gate is disabled. </del> <ins> // externalIP should have 1 (remote) endpoint since the feature gate is disabled. </ins> realServers2, rsErr2 := ipvs.GetRealServers(externalIPServer) assert.Nil(t, rsErr2, "Expected no error getting real servers") <del> assert.Len(t, realServers2, 0, "Expected 0 real servers") </del> <ins> assert.Len(t, realServers2, 1, "Expected 0 real servers") assert.Equal(t, realServers2[0].String(), "10.0.1.5:80") </ins> fp.OnEndpointSliceDelete(endpointSlice) fp.syncProxyRules()

<!-- Please only use this template for submitting enhancement requests --What would you like to be : In-cluster traffic for a Service loadbalancer IP with externalTrafficPolicy=Local should be masqueraded and routed to cluster-wide endpoints. A similar patch was to the iptables proxier in v1.15 in Why is this needed: Today, the LB IP traffic for the ipvs proxier is not differentiated from "Local" and "External" traffic, so in the case where a Service is using and traffic is originating in-cluster, the traffic will be black-holed if there are no local endpoints there.
/assign /sig network
/area ipvs
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/remove-triage unresolved
Patching kube-proxy with solves this issue.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I tried the proposed fix in From within a POD on a node where no server PODs are executing it works fine, with the source (POD) address preserved. From an external machine where incoming packets hit a node without server PODs (where it usually is black-holed) packets goes like this; On the k8s-node without server DNAT happens VIP -POD-address. The problem is that the reply takes a short-cut (much like a DSR) but the source is now the POD address (where it should have been nat'ed back to the VIP). The reply goes all the way back to the sender who have no idea about any connection with the POD-address as source and drops it. But if the external traffic setup is correct then external traffic should never hit any nodes where the server PODs are not executing. So IMHO this fix is OK.

Here is a trace from an external source showing the SYN to the VIP address (10.0.0.90) but the SYN-ACK has the POD address (11.0.4.3) as source:
Traffic from the main netns (e.g hostNetwork:True) from a k8s-node without server fails for a different reason than traffic from an external machine. Traffic from main netns on a k8s-node will get the VIP as source! This because the assign of the address to dev sets up a local route as; SYN is sent OK (e.g 10.0.0.90: -10.0.0.90:80) but the reply to the VIP address fails. This can be "fixed" by changing the route and set the node-ip as src; Then it works.
Is it possible to add the proposed solution in in kube-proxy? It would solve the problem for all users except for PODs with and have no significant draw-backs as far as I can tell.
I will take a look at that PR soon
Would you mind if I create a PR with your update described in ? Or would you prefer to do it yourself?
Hello. To solve this problem, I developed a kubernetes controller called node-network-manager. node-network-manager works on all kubernetes nodes and adds iptables DNAT rules from CLUSTER-IP to Load Balancer IP. Please try this and give me feedback. Thanks.
/area kube-proxy
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale There is a pending PR
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale There is a pending PR
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
hello. Please tell me. Did you manage to reach external-ip from host node ? Without a way to remove or replace the route and with src ip ? I have the same problems. When accessed from POD, it works ok. But when accessing from the host node to external-ip, it does not work. Replacing or removing the route from external-ip on the kube-ipvs0 interface helps. Is there any working solution for this?
No, sorry.

I think that the route must be replaced to make it work, as described in But that is a much more intrusive (i.e dangerous) update than the tiny fix in


Question: for idx, node := range nodes { caches, err := sysinfo.GetCacheInfo(sysFs, node.Cores[0].Threads[0]) if err != nil { <del> return nil, -1, fmt.Errorf("failed to get cache information for node %d: %v", node.Id, err) </del> <ins> glog.Errorf("failed to get cache information for node %d: %v", node.Id, err) continue </ins> } numThreadsPerCore := len(node.Cores[0].Threads) numThreadsPerNode := len(node.Cores) * numThreadsPerCore

I have some reports of a Node reporting a a resource capacity for cpu as a negative value. Any idea why cpu could report as a negative value? Is this an expected normal outcome? In its current state, it prevents any pods from being scheduled to that Node. Tips on how to debug further are appreciated. - any ideas?
Can you check machineInfo on Cadvisor or kubelet on the node?
a.k.a:
It is a bug in cAdvisor side. I am assigning it to you :-)
here is contents of /proc/cpuinfo Pertinent pieces of a log:
Machine is AWS instance if that matters....
We build the topology from system/cpu info. I think we can return partial topology information and ignore caches if its not present. This isn't a recent change though. Is it happening on just a single machine? I'll file a cAdvisor issue.
Can you link the referenced issue once created?
Re-assigning to since he's working on google/cadvisor
Could be this was caused by getting a Node on EC2 with old hypervisor, and I am the first victim...
That sounds likely. Most of the cache information is there in /proc/cpuinfo. So looks like the sysfs bits are disabled.
Can you try this cadvisor image on your machine and check /api/v1.3/machine output: sudo docker run  --volume=/:/rootfs:ro  --volume=/var/run:/var/run:rw  --volume=/sys:/sys:ro  --volume=/var/lib/docker/:/var/lib/docker:ro  --publish=8080:8080  --detach=true  --name=cadvisor  jnagal/cadvisor:canary
hmm
ah, selinux :( You'll need to run it with --privileged, I think. who's been dealing with it.
No luck with --privileged either on this box...
This has to do with SELinux, I think we had to disable it for this to work (some of our mounts are being denied). We're working on a better solution... This is not a problem in Kubernetes since we don't run Kubelet/cAdvisor in a Docker container.
ok, unfortunately, i have managed to kill docker on the box in question, and i cannot get the daemon to restart right now so I am not sure when that box will be available to debug against again.

I can provide an environment with a similar issue if required.
The fix is checked in to head. Can you build the binary and try?


Question: watcher := &cacheWatcher{ input:   make(chan watchCacheEvent, chanSize), result:  make(chan watch.Event, chanSize), <ins> done:    make(chan struct{}), </ins> filter:  filter, stopped: false, forget:  forget,

We noticed many goroutines stalled at in a large cluster (~200 nodes, ~1100 pods). It's because if the WatchServer timeout when the channel is full, will be blocked on the write to the even though is , because WatchServer stops consuming the channel after the timeout.
Great finding - thanks for debugging


Question: Spec: api.ServiceSpec{ Ports: []api.ServicePort{ { <del> Name:     "default", Protocol: api.Protocol("UDP"), Port:     14, </del> <ins> Name:       "default", Protocol:   api.Protocol("UDP"), Port:       14, TargetPort: util.NewIntOrStringFromInt(14), </ins> }, }, Selector: map[string]string{"func": "stream"}, Type:     api.ServiceTypeLoadBalancer, }, }, <del> expected: "you will also need to explicitly open a firewall", status:   200, </del> <ins> status: 200, </ins> }, }

This was broken by , and therefore broken in as well. Any fix should be offered as a cherry pick so "make release" is clean there.
I went ahead and worked something up for this this morning, to get Shippable builds back on the rails.


Question: } func (mounter *Mounter) IsLikelyNotMountPoint(file string) (bool, error) { <del> return true, nil </del> <ins> return true, unsupportedErr </ins> } func (mounter *Mounter) GetDeviceNameFromMount(mountPath, pluginDir string) (string, error) { <del> return "", nil </del> <ins> return "", unsupportedErr </ins> } func getDeviceNameFromMount(mounter Interface, mountPath, pluginDir string) (string, error) { <del> return "", nil </del> <ins> return "", unsupportedErr </ins> } func (mounter *Mounter) DeviceOpened(pathname string) (bool, error) { <del> return false, nil </del> <ins> return false, unsupportedErr </ins> } func (mounter *Mounter) PathIsDevice(pathname string) (bool, error) { <del> return true, nil </del> <ins> return true, unsupportedErr </ins> } func (mounter *Mounter) MakeRShared(path string) error { <del> return nil </del> <ins> return unsupportedErr </ins> } func (mounter *SafeFormatAndMount) formatAndMount(source string, target string, fstype string, options []string) error {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Calls to on unsupported platforms do not report any error and potentially misbehave. What you expected to happen: I would expect an error to be reported, that can be consumed by the caller. How to reproduce it (as minimally and precisely as possible): for and were failing on darwin, and no error was being reported. See: & Environment: OS (e.g. from /etc/os-release): I have been testing on osx/darwin, which is not a supported platform.
/sig storage
/assign


Question: "k8s.io/kubernetes/pkg/controller/petset" "k8s.io/kubernetes/pkg/labels" "k8s.io/kubernetes/pkg/runtime" <ins> "k8s.io/kubernetes/pkg/types" </ins> "k8s.io/kubernetes/pkg/util/sets" "k8s.io/kubernetes/pkg/util/wait" utilyaml "k8s.io/kubernetes/pkg/util/yaml" <ins> "k8s.io/kubernetes/pkg/watch" </ins> "k8s.io/kubernetes/test/e2e/framework" ) const ( petsetPoll = 10 * time.Second // Some pets install base packages via wget <del> petsetTimeout           = 10 * time.Minute </del> <ins> petsetTimeout = 10 * time.Minute // Timeout for pet pods to change state petPodTimeout           = 5 * time.Minute </ins> zookeeperManifestPath   = "test/e2e/testing-manifests/petset/zookeeper" mysqlGaleraManifestPath = "test/e2e/testing-manifests/petset/mysql-galera" redisManifestPath       = "test/e2e/testing-manifests/petset/redis"

Description of problem Create a petset, when pod is evicted, the petset can't create a new pod on other node and make sure pod is running. Kubernetes version (use ): v1.4.0-alpha.2.1427+ How to reproduce it (as minimally and precisely as possible): kubele eviction-hard="memory.available<${value}" a peset and wait all pod is running node MemoryPressure=true all pod status What happened: is evicted and petset can't create new pod and make sure pod is running. What you expected to happen: create new pod on other node and make sure pod is running.
Also when node come back with sufficient memory, petset can't make pod running again.
FYI
cc
Was able to reproduce it. Pet set controller just loops with the following logs I0830 12:32:52. 9 ] Syncing PetSet default/cockroachdb with 3 pets I0830 12:32:52. 9 ] PetSet cockroachdb blocked from scaling on pet cockroachdb-0 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-0 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-1 I0830 12:32:52. 9 ] PetSet cockroachdb waiting on unhealthy pet cockroachdb-2
-- do you have the petset YAML that we can evaluate? What is the quality of service tier for the pods produced by the petset? Can you include the of an evicted petset pod so we can see the final resource requirements spec?
I am able to reproduce it with nginx pet set in e2e test. Here is yaml output of it Also it seems that i have working fix and almost finished debugging e2e test (patchset is linked above). Will be grateful for review.
Ah, this helps clear things up... the linked pr shows the problem in petset controller and not kubelet...
Does it make sense to set milestone 1.4 for this issue? I think the fix is ready for review


Question: // waitLogs wait for the next log write. It returns two booleans and an error. The first boolean // indicates whether a new log is found; the second boolean if the log file was recreated; // the error is error happens during waiting new logs. <del> func waitLogs(ctx context.Context, id string, w *fsnotify.Watcher, runtimeService internalapi.RuntimeService) (bool, bool, error) { </del> <ins> func waitLogs(ctx context.Context, id string, logName string, w *fsnotify.Watcher, runtimeService internalapi.RuntimeService) (bool, bool, error) { </ins> // no need to wait if the pod is not running if running, err := isContainerRunning(ctx, id, runtimeService); !running { return false, false, err

won't output logs after kubelet rotate log filer of the container. kubectl output logs correctly after the log file is rotated. the config to . So kubelet will rotate the log soon. Build a image with this program a deploy with the previous image. the command No response <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/area kubelet
/triage accepted /priority important-longterm
/assign since you sent a PR already
Ran into this issue!


Question: } cmd.Flags().StringP("www", "w", "", "Also serve static files from the given directory under the specified prefix.") cmd.Flags().StringP("www-prefix", "P", "/static/", "Prefix to serve static files under, if static file directory is specified.") <del> cmd.Flags().StringP("api-prefix", "", "/api/", "Prefix to serve the proxied API under.") </del> <ins> cmd.Flags().StringP("api-prefix", "", "/", "Prefix to serve the proxied API under.") </ins> cmd.Flags().String("accept-paths", kubectl.DefaultPathAcceptRE, "Regular expression for paths that the proxy should accept.") cmd.Flags().String("reject-paths", kubectl.DefaultPathRejectRE, "Regular expression for paths that the proxy should reject.") cmd.Flags().String("accept-hosts", kubectl.DefaultHostAcceptRE, "Regular expression for hosts that the proxy should accept.")

Now that we've moved the experimental api to /apis/experimental... it's inaccessible from by default. We should fix that. who stepped on this earlier
security questions for you: this prefix was apparently in response to a security review quite some time ago. I don't understand how this adds much security. Can we just remove it? If not, is it acceptable to change the default from "/api/" to "/api", which would match both api locations? Otherwise we'll have to take a comma-separated list or something...
Ref
Oh, so this is not related to the security requests! And the original use case was just to re-root the URL path so it didn't interfere with static content. Sorry, I got confused because the last time I monkeyed with that, it was in conjunction with the allow/deny list that actually was the security request. In that case, let's see if we can change the default to "/", if that will break anything.
That change got introduced as part of to protect against XSRF in kubectl proxy. But I think the more important part of that change was that it disallows POST/PUT/PATCH by default. So as long as there is no state changing action outside of /api/ that can be triggered with a GET it should be safe to remove.


Question: matchTotal := int32(0) // nodeInfo.Pods() can be empty; or all pods don't fit for _, existingPod := range nodeInfo.Pods() { <del> if existingPod.Namespace != pod.Namespace { </del> <ins> // Bypass terminating Pod (see #87621). if existingPod.DeletionTimestamp != nil || existingPod.Namespace != pod.Namespace { </ins> continue } if constraint.selector.Matches(labels.Set(existingPod.Labels)) {

How to reproduce it (as minimally and precisely as possible): pre-requisites: a multi-az cluster, a deployment with 1 pod in each az. a with and . the node(s) in 1 az. the deployment's pod(s) from the cordoned az. What happened: A new pod was created and scheduled in a form that What you expected to happen: I expected that a new pod will be created and be "stuck" in status as it shouldn't have nowhere be scheduled. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ): 4.19.84-coreos Others: I created a with how I reproduce the bug
/sig scheduling
cc/
Thanks for raising this issue. I feel like it's a general timing issue - when you delete a Pod, that Pod may still exist for a very short time (in graceful deletion period), and at the same time, the controller backfills a Pod for it. So when scheduling tries to schedule the backfill Pod, it may read the cluster as: [1, 1, 1(deleting)]. To confirm this, can you try deleting the Pod using , and see if the problem goes away?
Yes sure I will try that. It sounds reasonable, thats why in the first demonstration I deleted all the pods and waited for all of them to completely disappear. Nevertheless, this is still something that I believe needs to be fixed. The pod's evaluation should take in consideration pods that are in status and don't count them as pods.
I do believe we exclude the Pods. (if not, it should be a serious regression). What concerns me is that at that moment, scheduler may treats that (deleting) Pod as running in its cache. I will try to debug into this.
Yep, this time the new pod was in status until I uncordoned the relevant nodes.


Question: return PodAdmitResult{Admit: true} } <del> // Always admit for remote runtime. if a.Runtime.Type() == kubetypes.RemoteContainerRuntime { </del> <ins> // Always admit runtimes except docker. if a.Runtime.Type() != kubetypes.DockerContainerRuntime { </ins> return PodAdmitResult{Admit: true} } <del> // Make sure it is either docker or rkt runtimes. if a.Runtime.Type() != kubetypes.DockerContainerRuntime && a.Runtime.Type() != kubetypes.RktContainerRuntime { </del> <ins> // Make sure docker api version is valid. rversion, err := a.Runtime.APIVersion() if err != nil { </ins> return PodAdmitResult{ Admit:   false, Reason:  "NoNewPrivs", <del> Message: fmt.Sprintf("Cannot enforce NoNewPrivs: %s runtime not supported", a.Runtime.Type()), </del> <ins> Message: fmt.Sprintf("Cannot enforce NoNewPrivs: %v", err), </ins> } } <del> if a.Runtime.Type() == kubetypes.DockerContainerRuntime { // Make sure docker api version is valid. rversion, err := a.Runtime.APIVersion() if err != nil { return PodAdmitResult{ Admit:   false, Reason:  "NoNewPrivs", Message: fmt.Sprintf("Cannot enforce NoNewPrivs: %v", err), } } v, err := rversion.Compare("1.23.0") if err != nil { return PodAdmitResult{ Admit:   false, Reason:  "NoNewPrivs", Message: fmt.Sprintf("Cannot enforce NoNewPrivs: %v", err), } </del> <ins> v, err := rversion.Compare("1.23.0") if err != nil { return PodAdmitResult{ Admit:   false, Reason:  "NoNewPrivs", Message: fmt.Sprintf("Cannot enforce NoNewPrivs: %v", err), </ins> } <del> // If the version is less than 1.23 it will return -1 above. if v == -1 { return PodAdmitResult{ Admit:   false, Reason:  "NoNewPrivs", Message: fmt.Sprintf("Cannot enforce NoNewPrivs: docker runtime API version %q must be greater than or equal to 1.23", rversion.String()), } </del> <ins> } // If the version is less than 1.23 it will return -1 above. if v == -1 { return PodAdmitResult{ Admit:   false, Reason:  "NoNewPrivs", Message: fmt.Sprintf("Cannot enforce NoNewPrivs: docker runtime API version %q must be greater than or equal to 1.23", rversion.String()), </ins> } }

I tried to add the support for , but the always failed with the error print: I checked the code of k8s, and found that "NoNewPrivs" is admitted only if the runtime is docker or rkt in kubelet. Related code is as follows: Maybe we should define a interface to get the capability of "NoNewPrivs" form runtime-shim.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig node
Sent to fix this. Thanks for finding this out!


Question: apiServerServiceTypeString       string apiServerServiceType             v1.ServiceType apiServerAdvertiseAddress        string <ins> apiServerNodePortPort            int32 apiServerNodePortPortPtr         *int32 </ins> apiServerEnableHTTPBasicAuth     bool apiServerEnableTokenAuth         bool }

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): FEATURE REQUEST <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): 1.6.2 Environment: Cloud provider or hardware configuration: AWS using Juju OS (e.g. from /etc/os-release): Ubuntu Xenial Kernel (e.g. ): 4.4.0-59-generic Install tools: Juju Others: What happened: kubefed init entered a infinite loop health checking the federation api service What you expected to happen: kubefed to exit with a success code How to reproduce it (as minimally and precisely as possible): Anything else we need to know: When doing a kubefed init on a cluster setup using VMs on aws (using Juju in my case) the worker nodes have two IPs: one that is visible to the internet and one internal visible only to VMs within aws. In my case for example the VM on aws hosting the kubernetes-worker node had external IP in the 54.174.73.1/24 subnet and it also had an internal IP in the 172.31.60.1/24 subnet. Using a client machine outside aws to bootstrap federation: I got the kubefed client stuck on doing a health check against the clouds internal subnet https://172.31.60.XXX:/healthz . To address this I can use the flag and pass the external IP of any kubernetes-worker node.

With this flag kubeconfig file gets updated with the right endpoint for the federation-apiserver. However, apart from the I should also be able to specify the desired port the service will be listening on. I have to make sure that port is opened on the aws routing/firewall prior to issuing . In the example above, kubefed init will try to reach https://54.174.73.XXX:/healthz, port is randomly assigned and is not reachable. I should be able to first open the port on aws and then do a with and port pinned to .
/cc
/assign
is there a reason why you are not using the default load balanced service on AWS?
I am using Juju to stay cloud agnostic. AWS just happened to be where I deployed this time. Juju just grabs machines (virtual or physical) from wherever you have resources from. In effect your cluster is more like an on-prem than anything else. How would the default load balanced service work?
got it.


Question: convert_api_ServiceSpec_To_v1_ServiceSpec, convert_api_ServiceStatus_To_v1_ServiceStatus, convert_api_Service_To_v1_Service, <del> convert_api_StatusCause_To_v1_StatusCause, convert_api_StatusDetails_To_v1_StatusDetails, </del> convert_api_Status_To_v1_Status, convert_api_TCPSocketAction_To_v1_TCPSocketAction, convert_api_TypeMeta_To_v1_TypeMeta,

When accessing a non existing entity (example - namespace), the message specifies the requested field as 'id', but it should be 'name'. Example (see the details section) { "kind": "Status", "apiVersion": "v1beta3", "metadata": {}, "status": "Failure", "message": "namespaces "staging" not found", "reason": "NotFound", "details": { "id": "staging", "kind": "namespaces" }, "code": 404 }
Yes. We should rename "id" to "name" in StatusDetails: Not sure if we can change v1beta3 API now. cc
You can add a new field and fill in both, and MAYBE remove ID in v1 , Nikhil Jindal wrote:
I suggest we fix it at most in v1 only. This is an annoying wart, but we could just fix the description string for now.
If trivial, we should fix v1.


Question: "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" semconv "go.opentelemetry.io/otel/semconv/v1.17.0" "go.opentelemetry.io/otel/trace" <ins> "k8s.io/apiserver/pkg/endpoints/request" </ins> tracing "k8s.io/component-base/tracing" )

When enable apiserver tracing, the webhook request spans only show spanName as /, mostly because use http request operations as default span name, which is hard to understand. ! Following the open-telemetry standard of spanName, it's recommended to use as spanName. ! SpanName of webhook request should be , like ! APIServerTracing feature gates of apiserver a pod tracing dashboard No response <details18 </details<detailsNone </details<details</details<details</details<details</details<details</details
I make a fix for this.
/sig api-machinery instrumentation
Do you know if this is fixed in later versions of opentelemetry?
/triage accepted /assign
Looks like might be related
not yet, I think we can fix this in otel upstream, but it's hard to backport to otel , which is the version of otel used in Kubernetes
Good point. It think fixing it here is a good idea in that case.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
could I take on this issue?
yes. Feel free to work on this.
! I test locally by 1.29 kubernetes and start a sample webhook, looks like the client span has already conforms to semantic conventions , which should be in the form of "HTTP {METHODNAME}"
! For apiserver as a server, now the spanName is still KubernetesAPI, which I think we could enhance it into "HTTP get_account/{accountId}" format
After digging for a while, I found one solution to implement this, I draft this idea in , could you help review about whether this is a correct way to implement it?


Question: if err != nil { log.Fatalf("Unable to create client; error: %vn", err) } <del> // Double check that worked by getting the server version. if v, err := client.Discovery().ServerVersion(); err != nil { log.Fatalf("Unable to get server version: %vn", err) } else { log.Printf("Server version: %#vn", v) </del> <ins> // Try to get the server version until <timeout>; we use a timeout because // the pod might not have immediate network connectivity. for start := time.Now(); time.Since(start) < timeout; time.Sleep(sleepTime) { // Double check that worked by getting the server version. if versionInfo, err = client.Discovery().ServerVersion(); err != nil { log.Printf("Unable to get server version: %v; retrying.n", err) } else { log.Printf("Server version: %#vn", versionInfo) break } time.Sleep(1 * time.Second) } if err != nil { log.Fatalf("Unable to contact Kubernetes: %vn", err) </ins> } for start := time.Now(); time.Since(start) < timeout; time.Sleep(sleepTime) {

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: N/A Which test(s) are failing: [] [sig-node] PreStop should call prestop when killing a pod [Conformance] Since when has it been failing: N/A Testgrid link: N/A Reason for failure: When spawning a Pod, it's container might start before all its networking plumbing has been completed. Thus, if the container tests / needs its network connectivity, it mgiht fail. The ` image tries to connect to the Kubernetes API, but if it cannot, the Pod will end up in an Error state. Retrying the Kubernetes API connection will avoid this issue. Anything else we need to know:
/sig node
/remove-sig node /sig testing


Question: return resultValue.Interface().(error) } <del> args := []reflect.Value{reflect.ValueOf(obj), reflect.ValueOf(options)} results := handler.printFunc.Call(args) </del> if results[1].IsNil() { rows := results[0].Interface().([]metav1alpha1.TableRow) printRows(output, rows, options) return nil } return results[1].Interface().(error) } // printRows writes the provided rows to output.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: 1when error has occurwe should not print header 2 when i get ns use --all-namespaces, i get NAMESPACE column, we should return error: storageclass is not namespaced What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
/sig cli


Question: gomega.BeTemporally(">=", time.Now().Add(-maxStatsAge)), // Now() is the test start time, not the match time, so permit a few extra minutes. gomega.BeTemporally("<", time.Now().Add(2*time.Minute))), <del> )})) </del> <ins> ), "Histogram": gstruct.Ignore(), })) </ins> }

There's been a quite a few failures across jobs that query node metrics. The test is Last passing - First failing - appears to point to which is bump in prometheus client library. Possibly related? /kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Verified with presubmit that reverting the prometheus client library results in the test passing. Opened PR for revert in
/cc
/assign
thanks for filing this issue! I think seems like a promising fix (thanks ) and there's a fair chance we won't need the revert
ack, thanks for the fix, it seems we can go with that then and hold the revert.


Question: terminationGracePeriodSeconds: 600 hostNetwork: true containers: <del> - image: gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.20.1 </del> <ins> - image: gcr.io/k8s-ingress-image-push/ingress-gce-glbc-amd64:v1.23.1 </ins> livenessProbe: httpGet: path: /healthz

master-informing gce-master-scale-correctnessKubernetes e2e suite.[It] [sig-network] Services GCE [Slow] should be able to create and tear down a standard-tier load balancer [Slow] ! 2023-05-24 10:01 PDT Kubernetes e2e suite: [It] [sig-network] Services GCE [Slow] should be able to create and tear down a standard-tier load balancer [Slow] No response /sig scalability cc
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
Ok, checking job Download controller-manager logs There are these errors already reported in but that doesn't seem the problem This is weird does this ring a bell? ^^
It seems to be a flake which is passed in the last run.
Yes, I see the last run Green; so closing this as of now! Will re-open it if this will be a release blocker.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This seems like a flaky test. It has failed multiple times: cc
/remove-kind failing-test /kind flake
The load balancer tkes a lot of time to be provisioned and the test times out I'm going to bump the ingress-gce version used on the test because is too old and has some performance improvements that can solve this issue
/reopen we need to be completely sure this address the problem, it has to get at least 5/5 consecutive greens
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
It failed again:
Checking last occurrence we can find the logs on the kube-controller-manager So, the problem is that the time to acquire the IP takes too much, the test times out at and the IP is at IIUIC this depends on the underlay infra, what do we want to do with this, bump the timeout of the LB?
/assign
Update on the state of failures. The failures are becoming more frequent: !
Unfortunately I think bumping the timeout is our only option.
can you send a PR to move to 1 hours and add a comment referencing this issue?
Just created
CI is green since 06-27. /close
Closing this issue. <detailsIn response to : CI is green since 06-27. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: FlushChain(table Table, chain Chain) error // DeleteChain deletes the specified chain.  If the chain did not exist, return error. DeleteChain(table Table, chain Chain) error <ins> // ChainExists tests whether the specified chain exists, returning an error if it // does not, or if it is unable to check. ChainExists(table Table, chain Chain) (bool, error) </ins> // EnsureRule checks if the specified rule is present and, if not, creates it.  If the rule existed, return true. EnsureRule(position RulePosition, table Table, chain Chain, args ...string) (bool, error) // DeleteRule checks if the specified rule is present and, if so, deletes it.

(update: likely this is benign, but it seems really weird when your reading kube proxy logs) in healthy, IPTABLES jobs, the various ip6tables-restore commands are happy....MEANWHILE in ipvs jobs (healthy, or non healthy, we see that some of the iptables dont exist which makes sense... why is ipvs kube proxy tyrying to do iptables-restore operations on the table ? why is it trying to do iptables stuff at all ? It seems like we get error messages when running ipv6. not sure why though? in particular i found these errors when looking at the test failures for You wouldnt see error messages for ip6tables-restore when running ipvs tests. This may be my own ignorance however, i.e. maybe this is normal. I guess every job has this (wether or not the job passes or fails) ? Here is an example from a passing test.... Similar logs from a failing tests that i originally saw this message ... has this failure, it is reproduced via the corresponding ci job ... the logs in the kube-proxy for one of the nodes look like this: ill add more links as i collect more data. Kubernetes version (use ): prow (main branch as of feb 12 2021)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
So, this happens in HEALTHY jobs to, so i guess its not a driver for an actual bug.... but its still weird to me :) ... updating original issue...
any idea on why those ipv6 errors are printed in ipvs code ?
The proxier is configured in dual-stack mode in the CI Since we enabled the dual-stack gate recently, this is the expected behaviour
I don't get similar printouts myself so I can't say anything for sure. By first check the old re-occuring (and very annoying) "KUBE-MARK-DROP" problem.

This chain is usually created by "kubeadm" but so far kubeadm is unaware of dual-stack and has only created this chain for ipv4 (in iptables). Then it needs to be created for IPv6 before kube-proxy is started; In my test cluster I always do this in a prep-stage before starting K8s.
The ci nodes doesn't have the kernel ip6tables nat modules loaded, hence they fail all the operation related
hmmm ok so is the issue here that we need to load that module in the CI jobs?
signing here because I want to at least find why ip6tables restore is called on kube-proxy ipvs things :D
ipvs does uses iptables too I think that kube-proxy should implement something similar to what implemented in kubelet The nat table for IPv6 was in 3.7 and it seems that it may be not present in some systems. /cc
/assign
I think it's mostly just about whether IPv6 is enabled, not about the kernel version.
IIRC this specific error in the CI is about the nat module only Anyway, you fixed it :clap: /unassign /assign


Question: }, expected: TopologyHint{ NUMANodeAffinity: NewTestBitMask(numaNodes...), <del> Preferred:        true, </del> <ins> Preferred:        false, </ins> }, }, {

What happened: I think I'm seeing a problem with the topology manager. I have two numa nodes, 22 cores per node. Kubelet has "--cpu-manager-policy=static --topology-manager-policy=single-numa-node ", and when I try to create a simple pod with 3 cpus and 500Mi of RAM in the guaranteed class it fails. Describing the pod gives "Resources cannot be allocated with Topology Locality", but I know there's lots of CPU and RAM available on both NUMA nodes. I'm instrumenting the topology manager kubelet code, currently there's not enough debug logs to actually be able to tell what's going wrong. What you expected to happen: I expect the pod container to be created on one or the other numa node and get allocated exclusive CPUs (via the static policy) on that NUMA node, and be affined to those CPUs. How to reproduce it (as minimally and precisely as possible): On a kubernetes node with multiple NUMA nodes, enable kubelet with static cpu manager policy and single-numa-node topology manager policy. Try to create a basic pod with a single container in the guaranteed QoS class. Anything else we need to know?: Environment: Kubernetes version (use ): sysadmin chris(keystoneadmin)]$ kubectl version Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"archive", BuildDate:"2019-10-09T05:41:44Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: wildcat pass hardware, 22 cores per NUMA node, two NUMA nodes. 64GB RAM per NUMA node. OS (e.g: ): CentOS 7 Kernel (e.g.

): 3.10.0-
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
oops, I thought the "close" meant the comment, not the whole issue. :)
I was able to reproduce this issue. The following (new) uint test in fails unexpectedly: With any policy other than the this test passes. This is due to 3 factors: 1) The CPUManager is the only hint provider passing actual valid hints 2) The devicemanager passes back as its hints, resulting in a single (i.e. don’t care) hint 3) We a special casing for the in the device manager that essentially turns the devicemanager's hint into a : Because the single hint from the devicemanager now has a preference, there is no chance to have it combine with any of the CPUManager hints to result in a successful hint. This should ultimately be addressed as part of the following in 1.17 But we should put a one-off patch into the 1.16.x branch to fix this in 1.16.
ptal
I've just updated with my observations. can you have a look and comment ?


Question: - apiGroups: - apps resources: <ins> - controllerrevisions </ins> - daemonsets - deployments - deployments/scale

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: User has role binding to in the namespace. What you expected to happen: To roll back successfully. How to reproduce it (as minimally and precisely as possible): A 1.10 cluster (haven't tested 1.11 but I suspect it's the same) and a 1.12 Anything else we need to know?: It seems that the built in and clusterroles don't cover , which is needed in =1.12 for to interact with revision history? Environment: Kubernetes version (use ):Install tools: Self-written ansible roles <!-- DO NOT EDIT BELOW THIS LINE --/kind bug /sig auth /sig cli /sig apps
opened and picks to 1.10, 1.11, 1.12


Question: kubernetes.io/os: linux containers: - name: metrics-server <del> image: k8s.gcr.io/metrics-server/metrics-server:v0.4.4 </del> <ins> image: k8s.gcr.io/metrics-server/metrics-server:v0.5.0 </ins> command: - /metrics-server - --metric-resolution=30s <del> # These are needed for GKE, which doesn't support secure communication yet. # Remove these lines for non-GKE clusters, and when GKE supports token-based auth. - --kubelet-port=10255 - --deprecated-kubelet-completely-insecure=true </del> <ins> - --kubelet-use-node-status-port - --kubelet-insecure-tls </ins> - --kubelet-preferred-address-types=InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP - --cert-dir=/tmp - --secure-port=443

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI -- (gce-ubuntu-master-default) Kubernetes e2e suite: BeforeSuite (There are other flakes in this job, but they are failing less often and different test each time, so let's say this issue is only for BeforeSuite) is not starting: Spyglass: Triage: Mentioned in /cc <!-- Please see the deflaking doc () for more guidance! --
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.22
/cc Since it is related to the metrics server. the metrics server keeps dying as the log shown in I cannot get more logs of metrics servers from the CI artifacts. Currently, we are using metric server 0.4.4 and the latest 0.5.0 is released. Are there some known bugs that may be related to this test failure?
I don't think so. /cc
The metrics-server fails to bind ~because there is another process listening~ on port 443 are some of the konnectivity pods listening on the port :443?
In the dockerfile of metrics-server if has ? Maybe it has permission problem?
ahh, right, nice catch, I didn't pay attention to the error and blindly assumed collision :sweat_smile: ... who can take an action on that? has this changed recently?
If the metrics-server version used after , I think it is correct. has no change recently.

I vaguely remember some changes recently to reduce privileges on pods on CI, tthis seems likely a permission problem, does this ring a bell? EDIT This PR introduced the change , any special reason why we want to run it in a privileged port if this is accessed using a Service? EDIT 2 let see if this is the problem
Yeah, Maybe you can refer to this, the release notes about v0.4.4 metrics-server . I think you have resolved this problem.
Release v0.3.X was already using privileged port 443, as v0.4.X has migrated to distroless image we needed to introduce to give it needed privileges. What's weird for me is that the test is flaking and not failing. Could it be related to node where MS is scheduled?
You mean, there may be other reasons? If so, may require in-depth analysis.
As far as I can tell, is never starting properly, even when the jobs are successful. If we were to take this job, that has succeeded: From the serial logs, we can see that the metrics-server pod is crash looping: But from the build logs, we can see that at some point the pod was deemed ready: To me it seems that there are 2 issues, one being that metrics-server doesn't have the capabilities it needs to run and another one that still needs to be investigated where metrics-server is sometimes marked ready even though it is in a crash looping state.
This is a problem with the metrics-server deployment on the CI jobs and the metrics tests I don't know which sig owns these components but it will be better to find first an owner, what is is clear is that the metrics tests are not reliable since once you set a readiness probe on the metrics-server it fails consistently , , however the tests pass cc
it seems the last try was the good one :smile:


Question: import ( dockertypes "github.com/docker/engine-api/types" <ins> dockercontainer "github.com/docker/engine-api/types/container" </ins> "k8s.io/kubernetes/pkg/api" <ins> kubecontainer "k8s.io/kubernetes/pkg/kubelet/container" </ins> ) <ins> // These two functions are OS specific (for now at least) func updateHostConfig(hc *dockercontainer.HostConfig, opts *kubecontainer.RunContainerOptions) { // no-op, there is a windows implementation that is different. } func DefaultMemorySwap() int64 { return 0 } </ins> func getContainerIP(container *dockertypes.ContainerJSON) string { result := "" if container.NetworkSettings != nil {

What happened: Latest kubelet + Windows Docker version 1.14.0-dev refused to start any containers, error output: "invalid option: Windows does not support MemorySwap" How to reproduce it (as minimally and precisely as possible): It's always reproducible with Windows Docker 1.14.0-dev + latest kubelet code base. Anything else do we need to know: It didn't happen for Windows Docker 1.12.2-cs2-ws-beta. The reason behind the issue was because new code was for Windows Docker to check Resources parameters, it doesn't accept a non-zero value for MemorySwap: But kubelet was default to set an unlimited (-1) value. The issue was gone in my setup after applied a private fix to disable MemorySwap by set it to zero for Windows.


Question: package metrics import ( <ins> "testing" </ins> "github.com/blang/semver" "github.com/prometheus/client_golang/prometheus" "github.com/stretchr/testify/assert" apimachineryversion "k8s.io/apimachinery/pkg/version" <del> "testing" </del> ) var (

The metrics framework requires the ability to optionally display hidden (deprecated) metrics. A boolean needs to be piped through the registry (possibly at initialization or possibly set later) which then needs to be passed to the Create function so that we can determine whether we need to actually create the metric. /sig instrumentation


Question: CROSS_BUILD_COPY qemu-QEMUARCH-static /usr/bin/ <del> RUN apt-get update && apt-get install -y --no-install-recommends stress &&  rm -rf /var/lib/apt/lists/* </del> <ins> RUN clean-install stress </ins> ADD consumer /consumer ADD consume-cpu /consume-cpu

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: BASEIMAGE for resource-consumer was ubuntu before and there is a ask for migrating to debian via - What you expected to happen: Use debian-base as BASEIMAGE for resource-consumer test image How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig testing


Question: gomega.Expect(result.Code).To(gomega.Equal(0)) } <del> ginkgo.By("Check node-problem-detector can post conditions and events to API server") for _, node := range nodes { ginkgo.By(fmt.Sprintf("Check node-problem-detector posted KernelDeadlock condition on node %q", node.Name)) gomega.Eventually(ctx, func() error { return verifyNodeCondition(ctx, f, "KernelDeadlock", v1.ConditionTrue, "DockerHung", node.Name) }, pollTimeout, pollInterval).Should(gomega.Succeed()) ginkgo.By(fmt.Sprintf("Check node-problem-detector posted DockerHung event on node %q", node.Name)) eventListOptions := metav1.ListOptions{FieldSelector: fields.Set{"involvedObject.kind": "Node"}.AsSelector().String()} gomega.Eventually(ctx, func(ctx context.Context) error { return verifyEvents(ctx, f, eventListOptions, 1, "DockerHung", node.Name) }, pollTimeout, pollInterval).Should(gomega.Succeed()) if checkForKubeletStart { // Node problem detector reports kubelet start events automatically starting from NPD v0.7.0+. // Since Kubelet may be restarted for a few times after node is booted. We just check the event // is detected, but do not check how many times Kubelet is started. // // Some test suites run for hours and KubeletStart event will already be cleaned up ginkgo.By(fmt.Sprintf("Check node-problem-detector posted KubeletStart event on node %q", node.Name)) gomega.Eventually(ctx, func(ctx context.Context) error { return verifyEventExists(ctx, f, eventListOptions, "KubeletStart", node.Name) }, pollTimeout, pollInterval).Should(gomega.Succeed()) } else { ginkgo.By("KubeletStart event will NOT be checked") } } </del> ginkgo.By("Gather node-problem-detector cpu and memory stats") numIterations := 60 for i := 1; i <= numIterations; i++ {

/kind failing-test <!-- If this is a flake, please add: /kind flake --/sig node
needs containerd 1.7.14 in the COS image - specifically
Thanks, I missed that. Checking the logs from i noticed that container is destroyed ! I see also from kernel logs that docker is blocked for 120second ( around the same time the container is destroyed ) Could it be that Docker Hung should be moved to be last in the test to avoid deleting a container?
/assign
/cc
Looking at the in job name initially I thought this job runs alpha features, so they might have forgotten to disable feature. But it seems this has already taken care -
/triage accepted


Question: expectedProtocol: "", expectError:      true, }, <ins> "no common protocol with comma separated list": { clientProtocols:  []string{"c, d"}, serverProtocols:  []string{"a", "b"}, expectedProtocol: "", expectError:      true, }, </ins> "common protocol": { clientProtocols:  []string{"b"}, serverProtocols:  []string{"a", "b"}, expectedProtocol: "b", }, <ins> "common protocol with comma separated list": { clientProtocols:  []string{"b, c"}, serverProtocols:  []string{"a", "b"}, expectedProtocol: "b", }, </ins> } for name, test := range tests {

What happened: I'm trying to establish a SPDY connection to exec into a pod. I sent this header: X-Stream-Protocol-Version: , , , The request failed with Forbidden - unable to upgrade: unable to negotiate protocol: client supports [, , , ], server accepts [ ] It appears that the server is not parsing the list out of the HTTP header and is treating it as a single value. What you expected to happen: Protocol negotiation should succeed. RFC2616 says: I can't simply send a multiple headers, because the HTTP library I'm using has no mechanism for sending separate headers as opposed to combining them. Kubernetes should parse the header whether it's given as separate headers or a list. Anything else we need to know?: I'm having to use SPDY because the web sockets implementation of exec . Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-05-16T16:14:56Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: Azure VMOS (e.g: ): NAME="Ubuntu" VERSION="16.04.6 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.6 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): Linux AdamArisDev 4.15.0-1063-azure -Ubuntu SMP Fri Nov 8 09:30:20 UTC 2019 x8664 x8664 x8664 GNU/LinuxOthers: My cluster is running in minikube v1.1.0.

/sig api-machinery
Thinking of the following change:
Since none of the accepted protocols include a comma, we should probably just iterate over all the headers, split by , trim each resulting item, omitting empty strings when computing clientProtocols, regardless of the number of headers.
Iterate over
Thanks for the quick update. I don't know what the expected HTTP behavior is when you use multiple headers and also have lists in each header, but if the same rule applies that multiple headers should be treated the same as one header with the items separated by commas, then presumably: Foo: a, b Foo: c, d should be the same as Foo: a, b, c, d in which case suggestion of basically moving the split into the inner loop would be more correct. In any case, I suppose a bug fix of this nature would only affect newer versions of Kubernetes? (I don't know if you do bug-fix-only patches to older versions.) I guess as a workaround I could try to catch the Forbidden error, look for the "unable to negotiate protocol" text, and switch to using only a single header value for older Kubernetes versions. Or is there a better approach?
That said, presumably this could also be handled in a lower layer - in the HTTP layer where headers are parsed - which might fix similar bugs with other headers that might exist elsewhere. (I think the only general exception to the equivalency rule is the Set-Cookie header because in practice cookies can contain commas but aren't quoted.)
That's correct. Given all the currently supported protocols are available in all supported kubernetes versions (the most recent protocols were in in v1.2.0 and in in v1.4.0), I'd probably just pick the specific protocol you want. If you wanted to detect a protocol failure, the X-Accepted-Stream-Protocol-Versions header is sent on a forbidden response due to protocol negotiation failure.
That's good to know. I can just stick to v4, then, since I don't expect to need anything that old. :-) Thanks.
/reopen I agree, should this fix be more general? Some other http headers have also the comma problems.
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.

</details
I would not recommend that, as not all headers value handling can safely assume commas are value delimiters.
I would keep this issue scoped to this particular header
/assign (since you are already working on this one, thanks!)
/close fixed in
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//pkg/apis/core/pods:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/errors:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/sets:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/validation/field:go_default_library", "//staging/src/k8s.io/apiserver/pkg/admission:go_default_library", <ins> "//vendor/k8s.io/klog/v2:go_default_library", </ins> ], )

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: We enabled AlwaysPullImages admission in an existing cluster, and found annotations/labels/finalizers of existing Pods whose imagePullPolicy configured as could not be updated. AlwaysPullImages admission mutated containerSpec by . And this change later got denied by Pod validation. It also caused terminating Pods with finalizers stuck forever until we removed AlwaysPullImages admission. What you expected to happen: Existing Pods are not affected by AlwaysPullImages admission controller. How to reproduce it (as minimally and precisely as possible): the kube-apiserver with AlwaysPullImages admission controller disabled a Pod with imagePullPolicy configured as in its containers AlwaysPullImages admission controller annotation to Pod Anything else we need to know?: If this is confirmed a bug, I can submit a PR to fix this. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig api-machinery
seems to be a bug. /assign
/remove-sig api-machinery /sig node auth
try to fix it in When AlwaysPullImages is disabled, you cannot update imagePullPolicy of pod. The error message is the same as yours. imagePullPolicy is immutable. When AlwaysPullImages is enabled, old pod(with imagePullPolicy=ifNotPresent) update will change imagePullPolicy as well. Workaround: recreated pod -_-
Thanks for the fixes. I still have regarding the PR. IMHO, it's OK that imagePullPolicy is not updatable. Why don't we just ignore Update in the admission, but choose to update spec instead? Any concerns?

This is another fix option. I'm not sure which is better. The pod will be still ifNotPresent after updating, even if the AlwaysPullImages is enabled. Which is expected behavior?
In my case, I would expect the admission not to touch existing Pods. Just want to make sure if there's any concerns.
I would recommend ignoring pod updates in AlwaysPullImages if the update does not change the set of image names referenced by containers in the pod. Changing images inline in a pod update is unusual, and that change would allow unrelated updates (to annotations, labels, etc) to proceed
Agree.
is updated .
This issue is fixed by


Question: if len(filteredNodes) == 0 { return result, &FitError{ Pod:              pod, <del> NumAllNodes:      len(nodes), </del> <ins> NumAllNodes:      numNodes, </ins> FailedPredicates: failedPredicateMap, } }

In : It seems that nodes are only used in the case there are no predicates. This call can be moved inside . Moving it there saves us from creating the node slice unless it is necessary. Checking if there are no nodes (line 202 below) can be done using . /kind cleanup /sig scheduling
I can work on this


Question: # returns a random port kube::util::get_random_port() { <del> awk -v min=1 -v max=65535 'BEGIN{srand(); print int(min+rand()*(max-min+1))}' </del> <ins> awk -v min=1024 -v max=65535 'BEGIN{srand(); print int(min+rand()*(max-min+1))}' </ins> } # use netcat to check if the host($1):port($2) is free (return 0 means free, 1 means used)

It seems like an issue similar to the problem with API server. I'll extend the timeout in the similar fashion. cc
Extending timeout didn't help. It's probably something else.
cc
I think the logs make it pretty clear what went wrong: or does one of you have time to fix this?
The Jenkins docker test image probably needs netcat to be installed. Should be an easy fix; I can take this.
Thanks!
Ugh... neverending story.
This seems like a different problem.
Oh, I see, this actually is the same, but harder: I think the most obvious thing to do is to let kubectl select the port and report back?
reports more occurances. Netcat somehow is no longer installed?
those two are older builds, using , which didn't have netcat yet.
ah thanks!
are you still working on this? Would you like me to take it?
I'm not right now, sorry, feel free to take it.


Question: } } <del> // Create Pod </del> <ins> t.Log("Creating the pod to schedule") </ins> if _, err := config.client.CoreV1().Pods(config.ns).Create(context.TODO(), test.pod, metav1.CreateOptions{}); err != nil { t.Fatalf("Failed to create Pod %q: %v", test.pod.Name, err) }

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestVolumeProvision Example: Since when has it been failing: Not sure, and aren't displaying any results /sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign /remove-lifecycle rotten Still seeing this, although not very frequently. Could indicate a race.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
latest failures:
take as an example the test fails b/c is set to PVC , which is not expected. our volume binder tries to dynamically provision it:
as for the test fails for the same reason when the scheduler volume binder tries to find volumes for PVC , its corresponding volume is not synced to volume binder or synced version is not in Available state so the volume binder will try to dynamically provision it too
I think the root cause is that the scheduler fails to find matching PVs in PV Assume Cache for the PVC because the PV in the Assume Cache is not in Available state or does not exist due to event delay. We should wait for the PV/PVC Assume Cache to be up to date before scheduling the pod.

However, it's not possible in integration tests. I think we can wait for the PV to be available or bound. This should reduce failures.


Question: Last updated 2017-06-06 04:24:44 UTC   Last updated 2017-06-08 02:30:43 UTC

Forked from We need to update the scheduler (ours and mesos) to use pods/binding instead of /binding and then we can delete /binding.
cc
This is about the top-level resource, not the subresource.
This is where mesos is using binding as top level resource: We need to change that to Resource("pods").Subresource("bindings")
/cc
If we turn this off in 1.2, will that be a problem for you?
Did we announce we were going to remove this API? Don't we need to have a process for taking it away?
Yes it would be problematic unless our scheduler was updated to use whatever was replacing it. AFAICT even the core scheduler is still using /bindings: , Clayton Coleman wrote:
Kicking this can down the road.
We should mark this deprecated.
According to , kube-mesos refactor will use HTTPExtender, so it's a not big issue to kube-mesos. But for k8s, I think scheduler is using API now ().
/assign
/close I'm going to close this; if any comments, please let me know.
We need to actually mark the endpoint deprecated so we can remove it.
OK, got that; need to deprecate it for other user. I think k8s can handle it to use .
, can you help to remove ? it's not related with Mesos anymore. I'll create a PR to deprecate .
Reopening since that PR just a deprecation warning. We still need to remove the API.
wow, agree. , would you help to remove milestone? We will not remove API in 1.7 or 1.8; but I'll trace that.
Done
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /lifecycle frozen
18 still have.
21 still have the resource. Is anyone already working on it or is there a reason why the binding resource still get served?

not entirely sure if i'm right, but from my current assumption it seems that there isn't an implementation for the for pods (xref: ) and we still use the generic resource for pods (). The resource is deprecated since 1.7 but i don't see any further progress :thinking: () dump of an :
what is the progress with this issue? Is it a backlog item that is tracked by sig scheduling?
I'm not aware of this issue. We will put it into our sig meeting to discuss.
we went through this in today's scheduling meeting. We're ok with shifting v1.Binding to pod's subresource. Before acting on it, there are several factors we need to consider: impact to external integrators - either 3rd-party scheduler or scheduler plugins (using extension point). Follow the conventional API deprecation/removal guide/process to ensure a smooth migration. overall volume of efforts. In addition to scheduling efforts, are there any API Server side efforts that need to be tweaked as well? we need to get +1 from
Is there a summary of why moving this API is necessary given that is has been around for quite sometime now? Is it related to RBAC?
/binding is a v1 API, and as such, deletion is not permitted. It can be marked deprecated, and users pointed to pods/binding by documentation as the preferred method, but not removed.
It's more to pay the technical debt. Given the action "binding" is performed against pods, composing a pod/binding instead of v1.Binding makes more sense. Thanks for clarification we cannot remove v1 APIs.
ok, thanks for the clarification, so this is not enabling a new use case, only a cleanup.
Right, and due to the stable API compatibility guarantees, I don't think it's a cleanup we can do. /close
Closing this issue. <detailsIn response to : so this is not enabling a new use case, only a cleanup. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Sorry for the late comment on this but i was AFK for a longer time. I'm not sure if we missed something. Currently there is only the implementation.

via the the API endpoint exists but it refers to the only existing kind (could be checked via ): As there isn't a seperate implementation for IMO we should remove the deprecation warning as we don't have an alternative implementation for that in place. / WDYT?


Question: "time" "k8s.io/kubernetes/pkg/api" <ins> apierrors "k8s.io/kubernetes/pkg/api/errors" </ins> "k8s.io/kubernetes/pkg/api/v1" legacyv1 "k8s.io/kubernetes/pkg/api/v1" metav1 "k8s.io/kubernetes/pkg/apis/meta/v1"

Three tests now create RoleBindings to give permissions to test service accounts: NodeProblemDetector Ingress PreStop Creating RBAC Rolebindings currently requires the user issuing the create call to have the required permissions. If RBAC is not enabled, and the cluster hasn't been configured with any RBAC bootstrapping, that won't be possible. We should make this test code conditional on whether RBAC is in use.
c.f.
Quick fix is to tolerate forbidden errors there and assume e2e setups using methods other than RBAC have already granted permissions to the service accounts
I'll put together a PR to tolerate the 403 in the test helper.
I think this can be closed now that the e2e RBAC setup is best effort and failures there don't affect the test if another authorizer is in use
I agree. My original thought was that we'd make the RBAC test stuff only happen on RBAC-enabled setups. But, since we're going to be pushing setups towards using RBAC, we might as well have those pieces stick around in the common code (as best-effort).


Question: ctx, cancel := watchtools.ContextWithOptionalTimeout(context.Background(), timeout) defer cancel() <del> preconditionFunc := func(store cache.Store) (bool, error) { _, exists, err := store.Get(&metav1.ObjectMeta{Namespace: ns, Name: name}) if err != nil { return true, err } if !exists { // We need to make sure we see the object in the cache before we start waiting for events // or we would be waiting for the timeout if such object didn't exist. // (e.g. it was deleted before we started informers so they wouldn't even see the delete event) return true, errors.NewNotFound(corev1.Resource("pods"), name) } return false, nil } </del> fieldSelector := fields.OneTermEqualSelector("metadata.name", name).String() lw := &cache.ListWatch{ ListFunc: func(options metav1.ListOptions) (runtime.Object, error) {

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: gce-master-scale-correctness Which test(s) are failing: [sig-cli] Kubectl client Simple pod should return command exit codes Since when has it been failing: 02/05 Testgrid link: https://k8s- Reason for failure: Anything else we need to know: /sig cli
/priority critical-urgent /assign
The test seems to be flaking quite a lot recently ! Friendly ping, sig-cli, could you take a look?
FTR, these are really blocking tests and it's getting more urgent given the forthcoming 1.18 release
Full log of the failure: It looks like the finishes with error. Strange, I thought is supposed to create a pod. Would really appreciate if someone from sig-cli could take a look
4 out of last 8 runs failed: https://k8s- /sig scalability
Another flaky, related test:
SIG CLI Leads -- any update on this? This is affecting scalability signal. /assign
-- Please keep tabs on this one.
yep, monitoring
It looks like there's a race between when the pod gets created and when we try to connect to it, thus we're seeing but further down in the logs the pod is , I'll be looking more into ti.
Thanks, Any update on this?
/cc
Sorry I got sidetracked yesterday and I'm back investigating this just now...
Probable fix in
Will continue to monitor here and reopen if needed :+1:
Unfortunately - it seems that the problem still exists - the test failed today (an hour ago): :( ^^
I'll check it, thx Wojtek for pointing that out.
- I'm not familiar with kubectl code, but I took a quick look and what I found is that this waitForPod() thing is happening only for pods with RestartPolicyNever: I must be missing something here, but for the case that failed it's not really the case, which means we don't wait for it?
This particular one is using , see the actual invocation that failed: That's why I started with this approach.
Sorry - the one for which the bug was originally opened.

But the today`s failure is actually slightly different (the same test, different case - sorry): The restart policy for this one is
Gotcha, so it looks like we progressed, I'll check the possible paths for this one, something tells me we're missing another timeout somewhere :wink:
OK - I think I know what is going on here. Here is what is happening (for the test case from two comments above): the test is setting "-i", so interactive mode is switched on that option triggers that Attach option defaults to true: over the Run() command, we get to the handleAttachPod: handleAttachPod is calling waitForPod And waitForPod is broken. The problem is the preconditionFunc: which is below passed to the UntilWithSync function UntilWithSync underneath is creating an informer and calling that function against its state: But the informer is initialized from apiserver watchcache, so it may be lagging. In large clusters we are observing lags in the order of small hundreds of milliseconds and that seem to be enough. What is happening is that: informer is initialized (but the pod creation is not yet reflected in it) the preconditionFunc is called the state doesn't contain that pod (yet) the whole thing fails with that error I actually believe, that just removing the preconditionFunc should actually solve the problem. I opened for that.
^^ - please take a look at the PR that I assigned to you (it's literally one-liner as of now - if you want to add tests or anything like that, feel free to pick it up).
Actually I went ahead, and fixed couple other commands too - so it's no longer one-liner. But still short one :)


Question: glog.V(vLevel).Infof("Image garbage collection succeeded") } <del> }, ImageGCPeriod, wait.NeverStop) </del> <ins> }, ImageGCPeriod, stopChan) </ins> } // initializeModules will initialize internal modules that do not require the container runtime to be up.

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The kubelet will GC the images on the node to free space when the used space reaching the percent ot the high-threhold(90% as default), but when the containers are not just managed by kubernetes(such as managed by swarm together) and not wished to be GCed by kubelet, I found we can not close the GC function by the flag of image-gc-high-threshold. What you expected to happen: add one switch to close the function of imageGC in specific senario: when the flag image-gc-high-threshold is set as 0, we should not GC the images on the node, just record one warning event to kube-apiserver Or add another switch flag to close the whole function of imageGC How to reproduce it (as minimally and precisely as possible): run kubernetes with several nodes, and test with making the inode all consumed of the nodes then the kubelet can triggle the GC to delete the oldest images on the node regardless the kind of them. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/kind feature
/sig node
What happens when --image-gc-low-threshold and --image-gc-high-threshold are both set to 100 ?
when both set to 100, image GC will still delete one image to release space as it always runs into the logic and delete image in logic of func
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle rotten /remove-lifecycle stale
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close


Question: options.captureProfile(outputProfile) } <ins> f.setInstrumentedPlugins() return f, nil } // setInstrumentedPlugins initializes instrumented plugins from current plugins that frameworkImpl has. func (f *frameworkImpl) setInstrumentedPlugins() { </ins> // Cache metric streams for prefilter and filter plugins. for i, pl := range f.preFilterPlugins { f.preFilterPlugins[i] = &instrumentedPreFilterPlugin{

metrics to see more details related to scheduler's decisions in Filter phase In some cases (e.g. new feature is in scheduler and want to see if it works or not), we may want to see which filter plugin affected Pod's scheduling results and it would be nice if we can observe that in metrics. Ref: /cc /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
So, your imagined metrics is like counter that has the label (i.e. one counter for one plugin) the count represents how many times did plugin have an impact on the scheduling of the pod. Am I correct? (I'm not sure if I am understanding your imagined metric correctly.)
I would also add an label that can be or . We probably need some common implementation between this and
do you have time to work on this in 1.26?
Yes, I'm gonna take time to work on this (+ ) in the next release. Let me try. /assign
We should probably extend this to Score as well
/retitle metrics to see which plugins affect to scheduler's decisions in Filter/Score phase
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale After supporting Skip in Filter and Score.
memo: is the issue to support Skip in Score.
/cc Will work on part after
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
It's time! /assign


Question: for _, volumeKindDir := range volumeKindDirs { volumeKind := volumeKindDir.Name() volumeKindPath := path.Join(podVolDir, volumeKind) <del> volumeNameDirs, err := ioutil.ReadDir(volumeKindPath) </del> <ins> // ioutil.ReadDir exits without returning any healthy dir when encountering the first lstat error // but skipping dirs means no cleanup for healthy volumes. switching to a no-exit api solves this problem volumeNameDirs, volumeNameDirsStat, err := util.ReadDirNoExit(volumeKindPath) </ins> if err != nil { return []*volumeTuple{}, fmt.Errorf("could not read directory %s: %v", volumeKindPath, err) } <del> for _, volumeNameDir := range volumeNameDirs { volumes = append(volumes, &volumeTuple{Kind: volumeKind, Name: volumeNameDir.Name()}) </del> <ins> for i, volumeNameDir := range volumeNameDirs { if volumeNameDir != nil { volumes = append(volumes, &volumeTuple{Kind: volumeKind, Name: volumeNameDir.Name()}) } else { lerr := volumeNameDirsStat[i] glog.Errorf("Could not read directory %s: %v", podVolDir, lerr) } </ins> } } return volumes, nil

When mounting several shares over NFS to container and when one of the directories is deleted from NFS server, something in the node is deleting content from other shares. I suspect that something goes wrong with "cleanupOrphanedVolumes"? not sure tough. Use case: php-app is running in container, uses php-sessions for remembering login. php sessions are stored in file and shared through nfs-share between containers. also app code is shared and several other directories to store different data (data dirs are mapped under code, code is in separate share so generated document files don't get to git tree), all is managed by separate integration server (in container), that uses same shares to prepare the code. This happened first time in GCE with Cluster API version 0.17.something, now it acts same on 1.0.1 and 0.18.2. Problem is repeatable: First of all you need to install package "nfs-common" to all cluster nodes, otherwise you can't map NFS shares at all: now you need nfs server. I created small instance for that with /etc/export stating: also I made directories under /export/test/: next you need to start it up (replace nfs server ip with your own): now start up the pod: verify, that the directories are mapped: now the funny part: remove directory from nfs server the zubuntu pod shows: nothing happens on nfs server yet, but just launch the stop command: and now nfs server shows: you see, that one file was left, but not for long. this is because root in client does not have root access in nfs-file system, but just after giving rights, the file is gone: now you can test "black hole" by creating new files. they dissaperar from the server in few moments. the dissapearing continues until cluster node is restarted.
so the problem looks to me is that containers delete files though they are not told to do. I am not quite convinced is the culprit here, since the directories are still alive. what the first version you believe this problem started to show up? Would it be possible for you to take a tcpdump on the nfs server so we can look at the nfs traffic? Thanks!
I started to play with the Google Container Engine about in middle of June and the version of the cluster was 0.17. something.

I don’t have that cluster anymore. So I don’t know what happened or did it worked before. About tcpdump. What ports and what traffic do you think you need for that? Also in what timeframe? Maybe you can even give me command-line for that? Jüri.
I am setting up an environment to simulate your tests, stay tuned
yes, I can reproduce your issue. And I was wrong about . You are more likely to be right on that suspect. Blame me if I point you to the wrong suspect. Here in this line , doesn't return any error if a volume fails to be torn down, which is what happening here: the first volume is not able to be torn down, yet the error message is suppressed, so later on thinks volumes are unmounted and it is safe to remove the directory
Something is fishy. Volume cleanup should be happening against the volume dirs, which is totally separate from pod dirs. Pod teardown should un-bind-mount the NFS from th epod dir, so pod cleanup can happen, even if the volume can't umount...
I didn't touch much of the volume code, but think the directory structure is . Removing pod directories would remove the volumes as well. We can delay the pod directory cleanup until all volumes in it are properly unmounted. Another user reported a similar case in
in this particular case, the mountpoint cannot be properly unmounted (because it was deleted), and I believe the current NFS mount is , so it won't be easy to unmount even after delay. unmount error might be a good heuristic. In this case, NFS stale handle error is probably good sign nothing is good for wait for.
Ahh yes, it was only GCE PD that was doing something tricker here , Yu-Ju Hong wrote:
I see. Thanks for the explanation. I didn't read the original post carefully. If we let propagate the error upstream, kubelet may always skip pod directory cleanup because of one failed teardown. That seems undesirable. I agree that the NFS stale handle error alone should be enough for us to skip waiting. In short, should distinguish the NFS error and report a successful teardown in this case. should check if there are any uncleaned volume directories, and avoid removing the specific pod.
cool, thanks.

When a patch is available, I'll give it a test on other volumes too.
This was completed by in PR can you take a crack at this?
ok, glad to do it.
Thanks!


Question: serv.Flags |= utilipvs.FlagPersistent serv.Timeout = uint32(svcInfo.StickyMaxAgeSeconds()) } <ins> // Set the source hash flag needed for the distribution method "mh" if proxier.ipvsScheduler == "mh" { serv.Flags |= utilipvs.FlagSourceHash } </ins> if err := proxier.syncService(svcPortNameString, serv, true, bindedAddresses); err == nil { activeIPVSServices[serv.String()] = true activeBindAddrs[serv.Address.String()] = true

The current support of the load distribution method and in kube-proxy ipvs mode only performs hashing of source IP address to select endpoints. As a result, all connections from a single client (i.e. from a particular source IP) are always forwarded to only one endpoint matching with the hash value of this source IP. If the client opens a vast majority of connections to the k8s cluster, it will result in traffic imbalance as only one endpoint located on one node handles the connections while the remaining endpoints/nodes are idle. Here is an example of the problem when using method for kube-proxy Deploy a service with 4 endpoints located on 4 nodes.Open 200 connections from a client to node IP of any node (e.g. telnet 192.168.1.1 ). All connections receive replies from only one single endpoint.The distribution printout also shows that all connections are forwarded to the endpoint (e.g. 1.0.4.2:5001 in this example)Similar issue is seen with method Connections from a client should be evenly distributed to all endpoints like to avoid unbalanced load distribution between cluster nodes and endpoints. This can be easily done by enabling the source port hash flag that IPVS supports for the methods which performs hashing on both source IP and source port when selecting the target endpoint.
/sig network /area kube-proxy /area ipvs
This issue was raised based on the discussion of the issue
triage/accepted /assign
/triage/accepted
/triage accepted (damn it!)


Question: "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/runtime/serializer/json" "k8s.io/apimachinery/pkg/runtime/serializer/versioning" <ins> "k8s.io/apimachinery/pkg/runtime/serializer/yaml" </ins> schedulerapi "k8s.io/kubernetes/pkg/scheduler/api" // Init the api v1 package _ "k8s.io/kubernetes/pkg/scheduler/api/v1"

Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". In latest scheduler componentconfig API (v1alpha1), it can be a file or a configmap: However, if you defines a scheduler config file and specify a YAML policy file: It doesn't work. It's because only JSON is supported: It'd nice to enhance it with an UniversalSerilizer so as to support both JSON and YAML. /sig scheduling /help /priority important-longterm
This request has been marked as needing help from a contributor. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Kubernetes scheduler enables you to overwrite default Predicates/Priorities as well as configuring extenders in a "scheduler policy". /help /priority important-longterm </details
/assign


Question: if len(nt.zones) == 0 { return "" } <ins> numExhaustedZones := 0 </ins> for { if nt.zoneIndex >= len(nt.zones) { nt.zoneIndex = 0 } zone := nt.zones[nt.zoneIndex] nt.zoneIndex++ <del> // We do not check the set of exhausted zones before calling next() on the zone. This ensures </del> <ins> // We do not check the exhausted zones before calling next() on the zone. This ensures </ins> // that if more nodes are added to a zone after it is exhausted, we iterate over the new nodes. nodeName, exhausted := nt.tree[zone].next() if exhausted { <del> nt.exhaustedZones.Insert(zone) if len(nt.exhaustedZones) == len(nt.zones) { // all zones are exhausted. we should reset. </del> <ins> numExhaustedZones++ if numExhaustedZones >= len(nt.zones) { // all zones are exhausted. we should reset. </ins> nt.resetExhausted() } } else {

Testgrid: https://k8s- Sample failure log: https://k8s- This test job has always been flaky, but is now failing more often than it's passing. Diagnosing it is difficult, though, because even the "successful" runs have multiple failures in the log, and the reported test failures seem not to be independant failures in the logs, but rather related to multiple failures to deploy pods and storage in the test setup. Does anyone have more insight into what's going on here? This feels more like a google cloud or test config failure than anything else. /kind failing-test /kind flake /sig test-infrastructure /sig gcp /sig apps
/sig testing
also looks like https://k8s- has similar issues, so this is actually not a gke specific problem.
one of the kubelet log has not sure if that's helpful at all... /assign (redirect to get some help from dev-oncall to see if can find any leads here...)
/assign /cc
GitHub didn't allow me to assign the following users: artemvmin. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The suites are failing on both gce and gke, so I don't think it's a gke specific issue. Looking at one occurrence: https://k8s- The first test that started failing was: Because it couldn't schedule the pod. Looking at the scheduler logs, I only see these two lines referring to that pod and nothing else: This is odd because there is usually at least one more log message if scheduling failed or succeeded. This implies that the pod is stuck somewhere in the scheduler and there could potentially be a deadlock. There are no other scheduling logs about other pods after this. An odd pattern that I noticed was that for all the serial runs where a ton of tests fail, the test case that always runs before the subsequent failing tests is the equivalence cache tests. And there's always a line in the like: I wonder if there is some race condition regarding the pod deletion case.

I have seen this pattern in the most recent runs where a bunch of tests all fail: https://k8s- https://k8s- https://k8s- https://k8s- The first failing test case is different across all these runs, but the commonality between them all is that the equivalence cache tests run right before it.
can you plz help take a look at this and help triage within scheduling?
fyi, this is pretty easy to repro on a dev cluster on head. I just run the EquivalenceCache tests, and then create any Pod afterwards. The Pod fails to schedule about 50% of the time.
I am investigating it.
I identified the issue and sent a PR to address it. We need to cherrypick the fix to 1.12 as well. Thanks to for the initial investigation and finding reproduction steps!
Thanks a lot and for looking into this and giving a fox PR. /cc needs to be CP'ed into 1.12 as well.
It already got approved. Please create a cherry pick PR for it.


Question: # Admission Controllers to invoke prior to persisting objects in cluster ENABLE_ADMISSION_PLUGINS="LimitRanger,ResourceQuota" <del> DISABLE_ADMISSION_PLUGINS="ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook" </del> <ins> DISABLE_ADMISSION_PLUGINS="ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,StorageObjectInUseProtection" </ins> # Include RBAC (to exercise bootstrapping), and AlwaysAllow to allow all actions AUTHORIZATION_MODE="RBAC,AlwaysAllow"

What happened: The preparation step of the e2e tests were failed sometimes on my local environment like What you expected to happen: These tests can be passed without any errors. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): 16.04.4 LTS (Xenial Xerus) Kernel (e.g. ): 4.4.0-138-generic Install tools: kubeadm Others:
/sig storage
After discussing on the PR I have known I need to enable StorageObjectInUseProtection plugin of admission controller which is disabled by default. The corresponding e2e tests require the plugin is enabled, so I think it is better to enable the plugin by default for most users.
Investigation of StorageObjectInUseProtection plugin of admission controller under kubernetes/kubernetes: cluster/centos/config- Enabled cluster/gce/config- Enabled cluster/gce/config- Enabled cluster/kubemark/gce/config- Enabled hack/local-up- Enabled cluster/juju/layers/kubernetes- Disabled This is the reason why we don't face this issue on the upstream CI system, and I it on my local environment which is deployed with kubeadm.


Question: continue } <ins> ginkgo.By("Removing the finalizer from the pod in case it was used") e2epod.NewPodClient(f).RemoveFinalizer(context.TODO(), p.Name, testFinalizer) </ins> framework.Logf("Deleting pod: %s", p.Name) e2epod.NewPodClient(f).DeleteSync(ctx, p.Name, metav1.DeleteOptions{}, 2*time.Minute) }

Pods that already succeeded and waiting for finalizer are being readmitted, fail to be admitted and moving from Succeeded Phase to Failed (admission failed) phase. The correct behavior would be to not try to re-admit the succeeded (or failed) Pod. Specific case: Node is exposing Devices using device plugin. Device plugin is implemented as a Pod Job's pod that is using the Device run to completion and waiting for Job Status finalizer kubelet is being restarted DevicePlugin pod is reconnecting to the kubelet. While it is reconnecting, kubelet don't have availability of devices kubelet re-admits the Pod, even though it is terminated already kubelet marks the pod as "failed to be admitted" But the same can happen with the variations of GC speed and other admission handlers. /sig node /priority important-soon /cc There is also a wider issue: , that concentrates on running pods. This issue is about terminated Pods, which is less controversial. <detailsFor sure on master, may be on earlier versions. </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
/triage accepted I think we have at least another issue mentioning the same behavior but observed for jobs, I'll look it up and crosslink later