Question: <ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package metrics import ( "sync" "time" "github.com/prometheus/client_golang/prometheus" ) const restClientSubsystem = "rest_client" var ( RequestLatency = prometheus.NewSummaryVec( prometheus.SummaryOpts{ Subsystem: restClientSubsystem, Name:      "request_latency_microseconds", Help:      "Request latency in microseconds. Broken down by verb and URL", }, []string{"verb", "url"}, ) ) var registerMetrics sync.Once // Register all metrics. func Register() { // Register the metrics. registerMetrics.Do(func() { prometheus.MustRegister(RequestLatency) }) } // Gets the time since the specified start in microseconds. func SinceInMicroseconds(start time.Time) float64 { return float64(time.Since(start).Nanoseconds() / time.Microsecond.Nanoseconds()) } </ins>

Currently we do not measure latency for REST operations. This is important for any component that performs non-watch operations (scheduler, controllers, kubelet). This is also prerequisite to actually make sure that all API calls take less than 1 sec ( ).
The apiserver is already , including for REST operations. You can scrape the metrics by hitting the /metrics HTTP endpoint.
Sorry, I was not clear in the description. I'm more interested on the client side and with more details so that I can measure things like: how long it takes to write bindings in scheduler how long it takes to update status in kubelet I'd think about it as a counterpart for apiserver stats.


Question: case v1.ResourcePods: r.AllowedPodNumber += int(rQuant.Value()) case v1.ResourceEphemeralStorage: <del> r.EphemeralStorage += rQuant.Value() </del> <ins> if utilfeature.DefaultFeatureGate.Enabled(features.LocalStorageCapacityIsolation) { // if the local storage capacity isolation feature gate is disabled, pods request 0 disk. r.EphemeralStorage += rQuant.Value() } </ins> default: if v1helper.IsScalarResourceName(rName) { r.AddScalar(rName, rQuant.Value())

The scheduler does not properly calculate pod requests for a resource as - it ignores init containers completely in . This would allow a workload to get "free" scheduling above the limit of containers (mitigated by quota, which calculates this properly) and so the scheduler would make non-optimal fits. Both kubelet and scheduler enforce the , so you can't actually schedule, but you sometimes get steered to the wrong node if you have more init container resource usage than containers. in should use and the similar helpers instead of writing its own method to calculate the request for resources. /sig scheduling
/priority important-soon
Let me follow this issue. /assign
fyi
After I checked the implementation of function, I think we just need to use it to instead of the function in the scheduler.
/priority critical-urgent
Is the pod phase not considered? When the pod phase is running, the requested resource is actually considered as . This change seems to reduce the resource utilization of the node.
Actually, I don't think we need to consider the pod.Phase. The resource request & limit are mainly used in the scheduling process. The scenario you said is not belong to the responsibility of kube-scheduler
Correct. Setting appropriate requests/limits that maintain good node usage is users responsibility. Considering the max(initContainers, sum(containers)) is the documented behavior.
I understand. Thank you for your comments!


Question: <ins> # This Dockerfile will build an image that is configured to use Fluentd to # collect container log files from the specified paths and send them to the # Google Cloud Logging API. # The environment variable that controls which log files are collected is # FILES_TO_COLLECT. Files specified in the environment variable should be # separated by whitespace, as in "/var/log/syslog /var/log/nginx/access.log". # This configuration assumes that the host performning the collection is a VM # that has been created with a logging.write scope and that the Logging API # has been enabled for the project in the Google Developer Console. FROM ubuntu:14.04 MAINTAINER Alex Robinson "arob@google.com" # Disable prompts from apt. ENV DEBIAN_FRONTEND noninteractive ENV OPTS_APT -y --force-yes --no-install-recommends # Install the Fluentd agent that knows how to send logs to Google Cloud Logging. RUN apt-get -q update &&  apt-get -y install curl &&  apt-get clean &&  curl -s https://storage.googleapis.com/signals-agents/logging/google-fluentd-install.sh | sudo bash # Copy the configuration file generator for creating input configurations for # each file specified in the FILES_TO_COLLECT environment variable. COPY config_generator.sh /usr/local/sbin/config_generator.sh # Copy the Fluentd configuration file for collecting from all the inputs # generated by the config generator and sending them to Google Cloud Logging. COPY google-fluentd.conf /etc/google-fluentd/google-fluentd.conf # Run the config generator to get the config files in place and start Fluentd. # We have to run the config generator at runtime rather than now so that it can # incorporate the files provided in the environment variable in its config. CMD /usr/local/sbin/config_generator.sh && /usr/sbin/google-fluentd -qq --use-v1-config --suppress-repeated-stacktrace > /var/log/google-fluentd/google-fluentd.log </ins>

We should provide a dockerfile for a pre-configured Fluentd image that can be passed file paths to collect logs from. We'll likely want one for exporting to elasticsearch and one for exporting to Google Cloud Logging, as we have today for the per-node stdout/stderr collectors. How to collect log files from within a container is a question that comes up occasionally (including ), and it provides a strong example of the usefulness of sidecar containers. cc
A controller that keeps one instance of a pod alive on each node would be useful for this. Shall I go ahead and write such a controller?
Would it be? This is intended to run as a sidecar container in a pod with the user's actual application. I think a pod-per-node controller is orthogonal (and is being discussed in and ). I'll create the actual Dockerfile, and a quick example of how to use it.
Sorry, my rush to catch up I think I mis-understood this thread.


Question: framework.Failf("%d errors:n%v", len(errs), strings.Join(messages, "n")) } }) <ins> }) framework.KubeDescribe("Pod Container lifecycle", func() { var podClient *framework.PodClient ginkgo.BeforeEach(func() { podClient = f.PodClient() }) ginkgo.It("should not create extra sandbox if all containers are done", func() { ginkgo.By("creating the pod that should always exit 0") name := "pod-always-succeed" + string(uuid.NewUUID()) image := imageutils.GetE2EImage(imageutils.BusyBox) pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: name, }, Spec: v1.PodSpec{ RestartPolicy: v1.RestartPolicyOnFailure, InitContainers: []v1.Container{ { Name:  "foo", Image: image, Command: []string{ "/bin/true", }, }, }, Containers: []v1.Container{ { Name:  "bar", Image: image, Command: []string{ "/bin/true", }, }, }, }, } ginkgo.By("submitting the pod to kubernetes") createdPod := podClient.Create(pod) defer func() { ginkgo.By("deleting the pod") podClient.Delete(context.TODO(), pod.Name, metav1.DeleteOptions{}) }() framework.ExpectNoError(e2epod.WaitForPodSuccessInNamespace(f.ClientSet, pod.Name, f.Namespace.Name)) var eventList *v1.EventList var err error ginkgo.By("Getting events about the pod") framework.ExpectNoError(wait.Poll(time.Second*2, time.Second*60, func() (bool, error) { selector := fields.Set{ "involvedObject.kind":      "Pod", "involvedObject.uid":       string(createdPod.UID), "involvedObject.namespace": f.Namespace.Name, "source":                   "kubelet", }.AsSelector().String() options := metav1.ListOptions{FieldSelector: selector} eventList, err = f.ClientSet.CoreV1().Events(f.Namespace.Name).List(context.TODO(), options) if err != nil { return false, err } if len(eventList.Items) > 0 { return true, nil } return false, nil })) ginkgo.By("Checking events about the pod") for _, event := range eventList.Items { if event.Reason == events.SandboxChanged { framework.Fail("Unexpected SandboxChanged event") } } }) </ins> }) })

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: For a Pod with , kubelet would attempt to create a new sandbox for it even after all container succeeded. Logs as below: This issue is similar to which fixed the case of pods whose RestartPolicy are Never. It caused unnecessary CRI and CNI calls, confusing logs and conflicts between the routine that creates the new sandbox and the routine that kills the Pod. It's likely also the root cause of because sometimes the routine that creates the second sandbox and the routine that kills unwanted pod run concurrently, then the netns would be gone when calling CNI. What you expected to happen: kubelet shouldn't create a new sandbox for pod with RestartPolicyOnFailure if all containers succeeded How to reproduce it (as minimally and precisely as possible): Create a pod like this: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.2 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Created which may fix this.
/assign


Question: "path/filepath" "strings" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> utilerrors "k8s.io/apimachinery/pkg/util/errors" utilvalidation "k8s.io/apimachinery/pkg/util/validation" utilfeature "k8s.io/apiserver/pkg/util/feature"

This one involves systemd, DNS, musl, and Kubelet v1.25 Starting in v1.25.0, pods running on Kubelets on certain nodes with a FQDN hostname (more on that later) will not be able to resolve DNS queries if they're using musl (e.g. alpine:3.16.2, busybox, etc). Our story starts with systemd. Back in 2020, systemd a "search ." in stub resolvers if it detected the hostname to be a FQDN (to solve a separate problem). Check for a static hostname with . In , a change was made to Kubelet to propagate that "search ." into pods (to solve a separate DNS problem). That change landed in and appears in . Unfortunatey, musl based DNS resolvers struggle with this "search ." EDIT: details in musl-based DNS resolvers (depite their other quirks) should continue to function To see if you have a suitable repro node, check your (likely a symlink to . Next, start a container (any CNI) that makes some use of musl resolvers. Confirm that starting in Kubelet v1.25.0, the container now includes the "search ." if it was present on the host. Try using musl-based DNS resolution. Note that uses its own DNS resolver, its not a reproducer. Note also that pod-to-pod traffic and outbound masquerade are fine too. This is not a masquerade problem. Edit the in the container to remove the "." and your musl problems instantly disappear. Add the dot and they're back. I've been spelunking for a while. I can confirm this is not related to a specific CNI - flannel, Calico, and Cilium all reproduce the issue. Not networking related OpenShift is not affected. In 2021, RedHat folks noticed DNS "breaks" and reverted the systemd change on their OS cc <details</details<detailsThis isn't cloud provider specific. However, the most likely cases where folks use FQDN's are on-prem imo. So you're likely to see reports originating from there.

</details<details</details<detailsThis was found while preparing Typhoon's v1.25.0 release of Kubernetes v1.25.0 </details<detailscontainerd 1.6.6 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
Could either revert (which propagates "search ." into containers) or find a different solution to the empty "search " problem it was trying to solve (does OpenShift even need that fix if they already ? cc A workaournd to problem is to configure hosts to strip "search ." from ala
has a great packet capture and this has been reported to musl
/sig network /cc
impressive investigation in the linked issue , that seems to conclude that is a muslc bug, since it happens outside of Kubernetes too. Just reverting the PR in Kubernetes doesn't looks right, it fixed a known bug ... and this is really a muslc bug .... maybe always removing the single dot from ? I think that for Kubernetes we should understand first what does and means and its implications, maybe some developer can help? you know someone?
Its discussed by the musl folks what means. It does not sound trivial to implement on the musl side, its a new case they did not anticipate supporting. Unfortunately this is subtle, and folks using alpine or related ecosystem things will get bitten and eventually end up here. I agree its a musl issue, but Kubelet now propagating triggers this I think its worth noting that did not set out to propagate , it was merely trying to prevent an empty . There may be better ways to achive that.
IIUC means "try the input name without any suffix". Resolvers will do this ANYWAY, but it's implementation-defined whether they do it first or last or in parallel, so allowing lets node owners be explicit (though, curiously, it seems at least glibc may try the same name AGAIN at the end of the search list).

This is pretty clearly a bug in MUSL, but I wonder if, instead of , we should have treated as rather than or the errant ? It nullifies the control that provides, but it wouldn't tickle musl like this.
/cc I foresee a new wave of bug reports caused because of this muslc bug , better to keep CoreDNS friends in the loop
The patch for this got merged for the upcoming 1.25.1 and 1.26.


Question: err = m.updateContainerCPUSet(containerID, cpus) if err != nil { glog.Errorf("[cpumanager] AddContainer error: %v", err) <del> return err </del> <ins> m.Lock() err := m.policy.RemoveContainer(m.state, containerID) if err != nil { glog.Errorf("[cpumanager] AddContainer rollback state error: %v", err) } m.Unlock() </ins> } <del> } else { glog.V(5).Infof("[cpumanager] update container resources is skipped due to cpu set is empty") </del> <ins> return err </ins> } <ins> glog.V(5).Infof("[cpumanager] update container resources is skipped due to cpu set is empty") </ins> return nil }

/kind bug What happened: Sometimes when an deployment creates new pods they get scheduled on a node successfully but the containers fail to start with: What you expected to happen: The scheduler shouldn't put pods on nodes without enough resources. How to reproduce it (as minimally and precisely as possible): I don't know how it got into this situation. Deleting the pods and have the RS recreate them works around this issue. Anything else we need to know?: We're using the CPUManager with static policy which might contribute to this problem. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): coreos /sig scheduling
/sig node (for CPU manager)
do the Kubelet logs seem similar to what's reported in this PR description?
/assign
Hi! :) No, I don't see these 'still running' messages. This is what I've got in the kubelet log (slightly redacted):
Can you share more information on this? Deployment manifest, platform details, frequency of these errors?
Fortunately we have open source the platform deployment: - but in short it's: AWS (c5.4xlarge instances for the workers where this issue is happening) multi master w/ kubeadm Enabled CPUManager and set to static policy Running 5 ASG: Controller, GPU-enabled workers and 3x regular workers (one per AZ) Using cluster autoscaler and HPA We have this problem with some pods pretty much all the time. Since we have pretty varying workloads, the CA often moves pods around and scales up/down which works around the problem by deleting the affected pods. So sometimes all pods are healthy, sometime maybe 15% of the pods have this issue. The biggest issue for us currently is that it's blocking rolling upgrades: Since new pods take forever to start, it blocks our CI. There are several deployments affected. All use HPA and all have resource limits of 2 cpus. One has 5Gi, the other 2Gi memory limits but that shouldn't matter. Here is one deployment manifest (slightly edited for internals) Why does this "Internal PreStartContainer hook" even exists? Shouldn't the scheduler be what knows about available resources and decides about this? Is there some doc that could help me understand how this is suppose to work?

I'd imaging if there is some limits the scheduler doesn't know about (like exclusively available cores or something), it would explain the issue but it would be unclear how this can work at all?
I've just realize that only one specific node (which is suppose to be the same as all other nodes) seems to be affected on which all pods that require full CPUs (triggering the static allocation) fail. Going to taint and drain the node now and will decomision it ~tomorrow, so let me know if you want me to pull something from the node first. Unfortunately I don't have long enough log rentention to see what kubelet might have logged before it got into this state.
My kubelet args:
Maybe relevant: Now that I'm trying to drain the node, I realize that there are ~100 argo () workflows objects "mapped" to the node. I don't know the details and how they are related to the node, I only know that I can't drain the node because of the workflows (that's another issue with argo or CRDs in general, see ). Maybe this is somehow blocking the cpus?
Ah, I misunderstood that error. There are ~100 completed pods that were created by argo. Could it by simply that somehow the finished pods don't free their cpu resources somehow?
We have fixed the completed pods related issue before:
would you mind digging into the cpu manager state checkpointed to disk? This will confirm or refute for us that we failed to release CPUs from a terminal container. The state file is located by default in and the content is simply utf-8 JSON. We should be able to see some container-to-cpuset mappings in there. If we can verify that some of the mappings are for terminal container IDs then that's where we should focus the search. My next question would be, were the PreStopContainer and PostStopContainer callbacks invoked, and if so, was there a failed write to the checkpoint file. PreStopContainer is not guaranteed to be called, in case the container process exits on its own. PostStopContainer is called as part of container GC. If there's some other problem interfering with the GC logic then that may have prevented cleanup of our CPU manager state too. Thanks for helping us to debug this remotely!

Ah damn, looks like the cluster autoscaler terminated that node already so unfortunately can't provide the cpumanagerstate.. Would be nice to have the relevant bits exposed as Prometheus metrics. I'll update this issue if this happens again.
"Good" news, problem just re-appeared - this time on two nodes. Here is one: cpumanagerstate: One of the containers doesn't exist at all and another one is exited: Here is the state field from for the exited container: It has "": "koko-intelligence--np2zh", set and according to k8s this pod is still running on that host: I found this in the kubelet log for this container:
Oh and it's indeed running, now in container which is also properly accounted for in the state file. Now regarding the non-existent container, I also see it in the kubelet log: Looks like this belongs to koko-gateway-track--6mljm, which doesn't exist anymore. But most likely because it just got gracefully removed eventually. Maybe some race condition preventing it to remove it and leading to these inconsistencies?
This is awesome info — thanks for updating. I won’t have time to look at this immediately, but perhaps can help trace it. I suspect some kind of race in reconcileState.
Thanks for the info I will take a look at this. /assign
/ Do you have a rough estimate for this? Fully understand if you have other things on your plate but then I'd probably revert the static policy since this becomes a bigger and bigger problem for us (it looks like it got worse over time, now even nodes are affected that were just brought up).
/ Is there some way I can help you debug/fix this? Seems unrelated to anything special in our setup from what I can tell. We had to disable this for all our deployments because it was causing severe issues.
/ Do you have any update/ETA on this?
Note that this is fixed in 1.12, but backports to 1.11 and 1.10 are not merged yet:
This still happens occasionally in 1.12 (with ) . provides cleanup for stalled entries.
Out of curiousity, does anyone still seeing this issue utilize containers in their pod Specs?

If so, the curlprit is likely addressed by the following:
The issue is still existing in kubernetes v1.14.4, without init containers to pod spec. Some way below may reproduce it: container crash on a node frequently and then delete the pod. a new pod and schedule to this node. Is this already fixed on master branch?


Question: }, Spec: api.ReplicationControllerSpec{ Replicas: 1, <del> Selector: podLabels, </del> <ins> Selector: rcPodLabels, </ins> Template: &api.PodTemplateSpec{ ObjectMeta: api.ObjectMeta{ <del> Labels: podLabels, </del> <ins> Labels: rcPodLabels, </ins> }, Spec: api.PodSpec{ Containers: []api.Container{

cc
Thanks for flagging it! Its not flaky, it is broken as we did not enable deployments for that test suite. Sent to fix that.


Question: deps = [ "//staging/src/k8s.io/api/authorization/v1beta1:go_default_library", "//staging/src/k8s.io/api/rbac/v1beta1:go_default_library", <del> "//staging/src/k8s.io/apimachinery/pkg/api/errors:go_default_library", </del> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/runtime/schema:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library",

What would you like to be : The function in should check for GKE and allow a missing SubjectAccessReview endpoint. If we get a 404 on another provider an error should still be returned. GKE doesn't enable the SAR endpoint. Without this endpoint, we cannot determine if the policy engine has adjusted as expected. In this case, we currently wait one second and hope it's up to date without returning an error and that it has adjusted as expected. Why is this needed: A 404 on that endpoint is not expected behavior for any provider other than GKE so it doesn't make any sense to implicitly allow 404s and hope that everything is ok. If it is a special, allowable, condition for GKE we should make that explicit in the code. The problem with doing it now is that checking the provider requires access to the TestContext which forces us to import the entire package. This goes against the goal of At the same time, we don't want to cause every test/caller of this to have to (1) know that GKE acts differently in this situation and code that one special path. This may be the lesser of the evils though, but regardless of how it is accomplished, at some point, the code should be validating this flow. Open to other solutions to this; maybe a specific test to test this code path (in which case this helper can be assumed to not be responsible for it?) Not sure if that is considered sufficient as it creates an awkward coupling between a helper function's assumptions and the existence of particular test.
/sig testing /area test /area test-framework
I think that comment no longer applies, and we can just remove that GKE special case. IIRC, GKE didn't initially enable serving the group, but it does now.
OK; I will: confirm GKE handling paste link here for history Put up PR removing the special case entirely


Question: "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" <ins> "k8s.io/apimachinery/pkg/labels" </ins> "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/types" <ins> utilrand "k8s.io/apimachinery/pkg/util/rand" </ins> "k8s.io/apimachinery/pkg/util/sets" "k8s.io/apimachinery/pkg/util/wait" <ins> "k8s.io/apimachinery/pkg/watch" </ins> clientset "k8s.io/client-go/kubernetes" <ins> "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch" </ins> "k8s.io/client-go/util/retry" batchinternal "k8s.io/kubernetes/pkg/apis/batch" "k8s.io/kubernetes/test/e2e/framework"

[x] Test approval issue : [x] Test PR : - Needed an update on the handling of anotations 27 March 2022 [x] Two weeks soak end date : 6 April 2022 [x] Test promotion PR : [x] Update Test PR: 15 April 2022 [x] Two weeks soak end date : 4 May 2022 with start of 1.25 [x] Update Test Promotion: According to these APIsnoop queries, there are still four remaining Job endpoints which are untested. Using a number of existing e2e test practices a new has been created for this job lifecycle test. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 4 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen for tracking e2e testgrid and promotion PR
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: go k.launchTask(driver, taskId, pod) } <del> // TODO(jdef) add metrics for this? type suicideTimer struct { timer *time.Timer } func (w *suicideTimer) Next(d time.Duration, driver bindings.ExecutorDriver, f jumper) suicideWatcher { return &suicideTimer{ timer: time.AfterFunc(d, func() { log.Warningf("Suicide timeout (%v) expired", d) f(driver, nil) }), } } func (w *suicideTimer) Stop() (result bool) { if w != nil && w.timer != nil { log.Infoln("stopping suicide watch") //TODO(jdef) debug result = w.timer.Stop() } return } // return true if the timer was successfully reset func (w *suicideTimer) Reset(d time.Duration) bool { if w != nil && w.timer != nil { log.Infoln("resetting suicide watch") //TODO(jdef) debug w.timer.Reset(d) return true } return false } </del> // determine whether we need to start a suicide countdown. if so, then start // a timer that, upon expiration, causes this executor to commit suicide. // this implementation runs asynchronously. callers that wish to wait for the

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Followed the documentation on rules to limit group Id's to non-superuser (root) using the rule. The creation attempt failed with: From PSP yaml: What you expected to happen: Successfully create a new PSP for non-superuser (root) group privileges using available rule values (which actually seems to be , , according the the ) How to reproduce it (as minimally and precisely as possible): Attempt to create a PSP with a rule set to Anything else we need to know?: I don't think so Environment: Kubernetes version (use ): , looks to affect , and doc as well Cloud provider or hardware configuration: IBM Cloud OS (e.g: ):Kernel (e.g. ): Install tools: Others:
/sig docs
/assign
thanks will send out a fix next week
I wonder whether this is a bug in implementation or the document. Is this really a documentation issue?
yes, RunAsGroup does not support MustRunAsNonRoot
Got it. Thank you!
/priority important-soon


Question: // Create the subPath directory on the host existing := path.Join(source.Path, subPath) <del> externalIP, err := framework.GetNodeExternalIP(&nodeList.Items[0]) </del> <ins> nodeIP, err := framework.GetNodeExternalIP(&nodeList.Items[0]) if err != nil { nodeIP, err = framework.GetNodeInternalIP(&nodeList.Items[0]) } </ins> framework.ExpectNoError(err) <del> result, err := framework.SSH(fmt.Sprintf("mkdir -p %s", existing), externalIP, framework.TestContext.Provider) </del> <ins> result, err := framework.SSH(fmt.Sprintf("mkdir -p %s", existing), nodeIP, framework.TestContext.Provider) </ins> framework.LogSSHResult(result) framework.ExpectNoError(err) if result.Code != 0 {

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: "[sig-storage] HostPath [It] should support existing single file subPath" test failed on environments which don't support external IPs like What you expected to happen: The test can pass successfully. How to reproduce it (as minimally and precisely as possible): just kick: Environment: Kubernetes version (use ): v1.11.1 Cloud provider or hardware configuration: OpenStack OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): 4.4.0-134-generic Install tools: kubeadm Others:
/sig storage


Question: } } <ins> func TestNestedDecodeStrictDecodingError(t *testing.T) { strictErr := runtime.NewStrictDecodingError([]error{fmt.Errorf("duplicate field")}) n := &testNestedDecodable{nestedErr: strictErr} decoder := &mockSerializer{obj: n} codec := NewCodec(nil, decoder, nil, nil, nil, nil, nil, nil, "TestNestedDecode") o, _, err := codec.Decode([]byte(`{}`), nil, n) if strictErr, ok := runtime.AsStrictDecodingError(err); !ok || err != strictErr { t.Errorf("unexpected error: %v", err) } if o != n { t.Errorf("did not successfully decode with strict decoding error: %v", o) } if !n.nestedCalled { t.Errorf("did not invoke nested decoder") } } </ins> func TestNestedEncode(t *testing.T) { n := &testNestedDecodable{nestedErr: fmt.Errorf("unable to decode")} n2 := &testNestedDecodable{nestedErr: fmt.Errorf("unable to decode 2")}

With the addition of server side validation in 1.23, it is now possible for a call to successfully decode it’s data and return a non-nil error when there are strict decoding errors (e.g. duplicate/unknown fields). This means that callers of can distinguish between fatal errors (that cause decoding to fail) and strict decoding errors (that still allow the decode to succeed) by inspecting the target object (to see if it was successfully decoded into) and the error itself (to see if it is a ) With the NestedObjectDecoder however, it only returns an error from giving no indication whether a strict decoding error it has returned comes from an internal decode step that short circuited (by not accounting for strict decoding errors) or actually did successfully decode, but also encountered strict decoding errors. We propose taking a few actions to minimize the possibility of erroneous short-circuiting from Document how all nested decoders should only return strict decoding errors if all decoding is successful or return only non-strict decoding (ie. fatal errors). Document this for all known in-tree and out-of-tree . Update in-tree nested decoders to do the above mentioned behavior (which is just the kube-scheduler) Long-term solutions to better guarantee that no erroneous short-circuiting is taking place could include deprecating the current interface in favor of one that returns both the target object that was decoded into along with an error return a special error type specific to indicating that all internal decoding was successful.
/sig api-machinery
/triage accepted


Question: # but we only need to rebuild targets if the contents actually changed.  That # is what the .stamp file represents. $(foreach dir, $(CONVERSION_DIRS), $(META_DIR)/$(dir)/$(CONVERSIONS_META)): <del> TAGS=$$(grep -h '^// *+k8s:conversion-gen=' $</*.go    </del> <ins> TAGS=$$(grep --color=never -h '^// *+k8s:conversion-gen=' $</*.go   </ins> | cut -f2- -d=                                     | sed 's|$(PRJ_SRC_PATH)/||');                     mkdir -p $(@D);

Using the grep from OSX, fails when GREPOPTIONS/GREPCOLOR is set as shown below (). Workaround : unset GREPOPTIONS/GREPCOLOR Note that using GNU grep using brew install will show that GREPOPTIONS shows immediately that GREP_OPTIONS may be interfering with make ()
oh, here's badly manged conversion-
Will look, thanks. For make-related bugs, please @ me directly.
Will do. thanks
This breaks building kubernetes inside Alpine Linux:
I'm pressing more and more on the idea that the only supported build is the dockerized build. Trying to accomodate every random environment is just too taxing. , Marc Villacorta wrote:
I agree with We can support the stuff in to run on mac to get you dockerized but everything else is only supported in there. I've worked around a few BSD/OSX things in but it isn't fun. We should try to keep that stuff isolated.
I agree, with the usual caveat that we need to be doing all of the build in one container with as little as possible happening outside. OS X tends to have old and broken versions of important programs, which is terrible. We discussed setting up CI for it but it's not going to happen in the near term. Alpine/Busybox purposefully omit features from their versions for good reasons, but it's still more work to support than it's worth.
Ok, I will re-think my workflow.
How did you find yourself building in Alpine, in I may ask? , Marc Villacorta wrote:
nothing is forcing me to use Alpine I just like the distro. I run stuff on CoreOS and my aim is to solve the build-deliver-run pipeline with a single small container. By the way, I have the problem installing the GNU package :-)


Question: func testMonitoringUsingHeapsterInfluxdb(c clientset.Interface) { // Check if heapster pods and services are up. <del> expectedPods, err := verifyExpectedRcsExistAndGetExpectedPods(c) framework.ExpectNoError(err) framework.ExpectNoError(expectedServicesExist(c)) // TODO: Wait for all pods and services to be running. </del> <ins> var expectedPods []string rcErr := fmt.Errorf("failed to verify expected RCs within timeout") serviceErr := fmt.Errorf("failed to verify expected services within timeout") err := wait.PollImmediate(sleepBetweenAttempts, initializationTimeout, func() (bool, error) { expectedPods, rcErr = verifyExpectedRcsExistAndGetExpectedPods(c) if rcErr != nil { framework.Logf("Waiting for expected RCs (got error: %v)", rcErr) return false, nil } serviceErr = expectedServicesExist(c) if serviceErr != nil { framework.Logf("Waiting for expected services (got error: %v)", serviceErr) return false, nil } return true, nil }) if err != nil { framework.ExpectNoError(rcErr) framework.ExpectNoError(serviceErr) framework.Failf("Failed to verify RCs and services within timeout: %v", err) } </ins> expectedNodes, err := getAllNodesInCluster(c) framework.ExpectNoError(err)

The following E2E test is failing in the GCE 1.5-1.6 upgrade jobs. Please take a look and let me know ASAP if this is a 1.6 blocker or not (I think it may be a flake, but need you to confirm). https://k8s- Expected error: <*errors.errorString | 0xc421d5ef10: { s: "expected to find one replica for RC or deployment with label heapster but got 0", } expected to find one replica for RC or deployment with label heapster but got 0 not to have occurred
/assign
can you help get this looked at and triaged ASAP? Thanks.
I pinged by email. He's really the one who should look at this.
The problem seems to be not with the test itself but with Kubernetes or test framework, since the test failed on starting Heapster. I'll investigate more later today.
I haven't got time to investigate. This seems to be unrelated to monitoring itself, so any help during MTV day would be great. If not I'll try to fix this tomorrow
thanks
The problem is in , in function . The function is called immediately from , w/o waiting for system pods (heapster) to start. We should give some time for heapster to become ready, as it takes few minutes after cluster creation.
CC
thanks a lot for investigation! is going to fix it.


Question: t.Errorf("autoSubnetInList() = %v, expected: %v", gotNames, expectedNames) } } <ins> func TestFirewallToGcloudArgs(t *testing.T) { firewall := compute.Firewall{ Description:  "Last Line of Defense", TargetTags:   []string{"jock-nodes", "band-nodes"}, SourceRanges: []string{"3.3.3.3/20", "1.1.1.1/20", "2.2.2.2/20"}, Allowed: []*compute.FirewallAllowed{ { IPProtocol: "udp", Ports:      []string{"321", "123-456", "123"}, }, { IPProtocol: "tcp", Ports:      []string{"321", "123-456", "123"}, }, }, } got := firewallToGcloudArgs(&firewall, "my-project") var e = `--description "Last Line of Defense" --allow tcp:123,tcp:123-456,tcp:321,udp:123,udp:123-456,udp:321 --source-ranges 1.1.1.1/20,2.2.2.2/20,3.3.3.3/20 --target-tags band-nodes,jock-nodes --project my-project` if got != e { t.Errorf("%q does not equal %q", got, e) } } </ins>

/kind bug /sig network /assign What happened: For events being generated within the GCE cloudprovider, slices of firewall parameters are not being sorted before converted into a string. Due to random ordering, this may result in the same line being repeated.


Question: package app import ( <del> "github.com/prometheus/client_golang/prometheus" </del> "net/http" goruntime "runtime" <ins> "github.com/prometheus/client_golang/prometheus" </ins> genericapifilters "k8s.io/apiserver/pkg/endpoints/filters" apirequest "k8s.io/apiserver/pkg/endpoints/request" apiserver "k8s.io/apiserver/pkg/server"

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: // Log is intended to be called once at the end of your request handler, via defer func (rl *respLogger) Log() { latency := time.Since(rl.startTime) <del> glog.V(2).Infof("%s %s: (%v) %v%v%v", rl.req.Method, rl.req.RequestURI, latency, rl.status, rl.statusStack, rl.addedInfo) </del> <ins> if glog.V(2) { glog.InfoDepth(1, fmt.Sprintf("%s %s: (%v) %v%v%v", rl.req.Method, rl.req.RequestURI, latency, rl.status, rl.statusStack, rl.addedInfo)) } </ins> } // Header implements http.ResponseWriter.

the file and line for these log statements is in the That's not helpful I1031 11:09:06. ] 'foo' has no storage object I1031 11:09:06. ] 'pods' is not a redirector I'd rather see Maybe use runtime.Caller in httplog to somehow fake the log to come from the caller's file/line.
Yeah-- actually this should be a feature of the glog package. Would be awesome to do:
Followup: glog is an opensource copy of a google-internal library; there's already something enabling this internally; we'll try to get a new version pushed externally in the next week or so. Then we can fix this.
Awesome. On Oct 31, 2014 2:53 PM, "Daniel Smith" wrote:
should meet your requirements.
+1, I would love to have this.


Question: <ins> /* Copyright 2017 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package apiserver import ( "testing" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" ) func TestConvertFieldLabel(t *testing.T) { tests := []struct { name          string clusterScoped bool label         string expectError   bool }{ { name:          "cluster scoped - name is ok", clusterScoped: true, label:         "metadata.name", }, { name:          "cluster scoped - namespace is not ok", clusterScoped: true, label:         "metadata.namespace", expectError:   true, }, { name:          "cluster scoped - other field is not ok", clusterScoped: true, label:         "some.other.field", expectError:   true, }, { name:  "namespace scoped - name is ok", label: "metadata.name", }, { name:  "namespace scoped - namespace is ok", label: "metadata.namespace", }, { name:        "namespace scoped - other field is not ok", label:       "some.other.field", expectError: true, }, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { c := crdObjectConverter{ UnstructuredObjectConverter: unstructured.UnstructuredObjectConverter{}, clusterScoped:               test.clusterScoped, } label, value, err := c.ConvertFieldLabel("", "", test.label, "value") if e, a := test.expectError, err != nil; e != a { t.Fatalf("err: expected %t, got %t", e, a) } if test.expectError { if e, a := "field label not supported: "+test.label, err.Error(); e != a { t.Errorf("err: expected %s, got %s", e, a) } return } if e, a := test.label, label; e != a { t.Errorf("label: expected %s, got %s", e, a) } if e, a := "value", value; e != a { t.Errorf("value: expected %s, got %s", e, a) } }) } } </ins>

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: got an error when trying to use a fieldSelector when watching a custom resource What you expected to happen: no error How to reproduce it (as minimally and precisely as possible): a CRD and then a CR for it. to watch with a fieldSelector, such as Anything else we need to know?: The customresource implementation has a defined, and it looks like it should support at least a field selector for and , but I'm guessing we need a bit more in place to make this work for . Environment: Kubernetes version (use ): HEAD (commit ) /sig api-machinery I can try to submit a PR to fix once I finish my current work item, if nobody picks this up sooner.
/area third-party-resource
cc
This is a more general form of . I don't have immediate plans to work on either of them, so you're welcome to take it.
sounds good. Do you want me to close this as a dupe?
I think there's a useful distinction between the two issues. If we support only , that would be enough to but this one tracks the idea of supporting arbitrary field selectors. By the way, if I recall correctly, when I applied the hack mentioned in the other thread to skip the step for CRDs, I think I saw that actually arbitrary field selectors worked too (not just ). That would mean enabling this might be easy on a technical level, although we may need to think through whether this could tie our hands if we ever wanted to add versioning support for CRD in the future. This issue is a better home for that discussion than the other one.
This would help a lot with debugging in prow. Today, the only thing we can do is bubble up information from the spec of a prowjob as labels and use a label selector.
What about just having 's just return the passed in label and value without any "conversion" for now?
I don't think we'd want to do it for all Unstructured, since there's no guarantee all Unstructured objects are CRD objects. However, it should be pretty easy to arrange for ConvertFieldLabel to be a no-op for CRD objects, which makes sense currently because CRD only allows one version for a given resource.

We just need to make sure this will not prevent us from doing more with CRD versioning in the future. My suspicion is that it should be fine, but I haven't investigated properly.
ok, I can put together a PR to make this change. any gut reactions?
PR:
I am sorry to comment on this closed thread. Currently, I am working on a CRD and needs to support arbitrary field selectors. I've tried to do it by learning from what you do for . But it always throws error: . So, I am wondering if I am doing something wrong or it only supports limited field selectors for CRD, which is stated . Here is my code snippet:
CRDs still only support namespace and name for field selectors
Any plan to support arbitrary field selector for CRDs?
Not currently

Follow a TPR an instance a single instance return 400. (watch a collection of TPR works) master and kubectl version: 1.7.1 CRD has the same issue. /assign
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
I was able to reproduce this with CRD on master. The problem is that kubectl implements a watch on a single object as a watch on the collection with a field selector that filters by . However, field selectors don't work on CRD objects because they are stored as , which does not support : This method is used to transform the field selector from versioned to internal field names: As a test, I bypassed that check for Unstructured objects, and watches on single CRD objects started working. Given that CRD doesn't support multiple versions anyway, it seems feasible to support this with a properly-targeted override of specific to CRD objects. cc
If we could support name and namespace generally on all objects containing object meta, I think I'm in favor of the concept.
I think it's feasible to fix this for CRD at some point. However, given that TPR is already gone in master, I don't think it's worth fixing for TPR unless it's somehow a regression in 1.7. Do you have any objections to that?
Having CRD support is good enough for me and service catalog. Will you backport the fix for CRD to 1.7?
At this stage I wouldn't expect that since it isn't a regression (I don't think) for either extension mechanism.
cc
I am also getting the same error when i try to watch CRDs and filter on a specific field. Im a currently using the to watch for CRDs. On a related note, i noticed that the takes a . Is there a way to pass a label selector so that i can watch CRDs using labels?
Also is this issue slated for 1.8?

(For CRDs that is)
The general issue of supporting field selectors for CRD is tracked in . I don't think it's slated for 1.8. The methods on should take a , which can contain an optional .


Question: errorType string test      string }{ <del> /*{ </del> <ins> { </ins> pod: &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:   "fakename",

A significant number of the of are commented out, possibly was done by mistake in the past. I uncommented the tests to see if they pass, but one of them failed. What we need is to uncomment those tests, fix the one that is not passing and remove the redundant ones if there are any. /sig scheduling /help
/assign I will be moving some of the tests to unit tests and leave a cleaner set of integration tests.
/remove-help
would you like to take this? I can provide guidance.
/help /unassign
/assign
/remove-help


Question: var _ volume.ProvisionableVolumePlugin = &azureDataDiskPlugin{} var _ volume.AttachableVolumePlugin = &azureDataDiskPlugin{} var _ volume.VolumePluginWithAttachLimits = &azureDataDiskPlugin{} <ins> var _ volume.ExpandableVolumePlugin = &azureDataDiskPlugin{} </ins> const ( azureDataDiskPluginName = "kubernetes.io/azure-disk"

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind feature What happened: According to , some volumes(e.g. aws, nfs, ceph) has already supported PV resizing, need to add PV resize support for azure disk What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: PV resize doc is here: Update: For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I will try to work on resizing an azure disk which is not in attached state. May dig into more details later. Environment: Kubernetes version (use ): v1.10.0 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign
Hi , is PV size grow feature still in alpha in master now? How do I open this feature, set in kubelet, kube-apiserer and kube-controller-manager config?
I have figured out that should be set in kubelet, kube-apiserer, kube-controller-manager, kube-scheduler config, and also need to enable in apiserver admission control. It works well.
should we go ahead and close this issue?
also file needs to be updated to allow resizing. This plugin provides additional checks for resizing.
For azure disk, there is blocking issue: it does not support disk resize on a Running VM. I would suggest work on first, could you review , it works well in my testing env.
Adding priority important-soon to this issue so that it can stay in 1.10. If that is incorrect, please update. priority important-soon
/priority important-soon
PR has been merged, can this issue be closed?
yup, looks like it's in. Thanks for flagging! /close
you can't close an active issue unless you authored it or you are assigned to it, Can only assign issues to org members and/or repo collaborators.. <detailsIn response to : yup, looks like it's in. Thanks for flagging! /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Awww shucks, looks like I don't have magic powers. please close this when you have a chance?
has been merged, while for azure disk work is just in investigation state mainly because there is blocking issue: it does not support disk resize on a Running VM.
could someone move this issue out of milestone v1.10, thanks.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Why was this closed? This is not fixed by that crappy PR. This is the kubernetes repo, and this refers to k8s apis being implemented correctly iyam. The PR that you closed this with does NOT support disk resize on a Running VM. Your words Please reopen this.
pls track this issue for online resize feature:


Question: if info.IsDir() { mask |= os.ModeSetgid <ins> mask |= execMask </ins> } err = os.Chmod(path, info.Mode()|mask)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I am using Persistent Local Volumes. The mountpoints for it are set up with 0700 permissions. When running a pod with a user other than root, and with fsGroup to allow access to the volume, the permissions are wrong and the pod cannot use the volume. The permissions are changed from 0700 to 2760. What you expected to happen: fsGroup makes the volume usable for the group by setting the permissions to 2770. How to reproduce it (as minimally and precisely as possible): You need a cluster with local volumes whose permissions are set to 0700. pv-permission- As you can see, the permissions are 2760. Since the execute bit is missing for the group, the directory cannot be used. Anything else we need to know?: Environment: Kubernetes version (use ): v1.11.2 Cloud provider or hardware configuration: libvirt/QEMU OS (e.g. from /etc/os-release): CentOS Linux release 7.5.1804 (Core) Kernel (e.g. ): 3.10.0-862.11.6.el7.x8664 SMP Tue Aug 14 21:49:04 UTC 2018 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others:
/sig storage
The fix is probably to add execute bits here
do you see any problem with setting group execute for fsgroup?
I think that execute should be fine.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten


Question: // acceptedNames are the names that are actually being used to serve discovery. // They may be different than the names in spec. <ins> // +optional </ins> AcceptedNames CustomResourceDefinitionNames `json:"acceptedNames" protobuf:"bytes,2,opt,name=acceptedNames"` // storedVersions lists all versions of CustomResources that were ever persisted. Tracking these

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Currently, the field on must be explicitly set to their zero-values due to the fact that they are not tagged . Ref: Why is this needed: If using a system that automatically compares the provided configuration to the current, it will produce a diff on the updated status after creation. In addition, making sure to explicitly set to zero-values is not the best user experience. I am happy to provide the fix if this change is approved! /cc /sig api-machinery /area custom-resources
marking them as seems ok, so that client-side validation doesn't complain if they are absent when submitting manifests, but they shouldn't need to be set to omitempty (and at this point, we likely wouldn't change the serialization) Those fields are changed by controllers after creation, so expecting the live API object to exactly match the status is not going to go well anyway.
Good point! I’ll open a fix for this. Yep! The specific system in question ignores fields that are not set at creation time by default when doing comparison, so if status fields are not included then it will not report a diff.
/assign
I opened a fix for this but CC'ed both you and Let me know if any additional effort is required. Thanks!


Question: if err := o.drainer.DeleteOrEvictPods(list.Pods()); err != nil { pendingList, newErrs := o.drainer.GetPodsForDeletion(nodeInfo.Name) <del> fmt.Fprintf(o.ErrOut, "There are pending pods in node %q when an error occurred: %vn", nodeInfo.Name, err) for _, pendingPod := range pendingList.Pods() { fmt.Fprintf(o.ErrOut, "%s/%sn", "pod", pendingPod.Name) </del> <ins> if pendingList != nil { pods := pendingList.Pods() if len(pods) != 0 { fmt.Fprintf(o.ErrOut, "There are pending pods in node %q when an error occurred: %vn", nodeInfo.Name, err) for _, pendingPod := range pods { fmt.Fprintf(o.ErrOut, "%s/%sn", "pod", pendingPod.Name) } } </ins> } if newErrs != nil { <del> fmt.Fprintf(o.ErrOut, "following errors also occurred:n%s", utilerrors.NewAggregate(newErrs)) </del> <ins> fmt.Fprintf(o.ErrOut, "Following errors occurred while getting the list of pods to delete:n%s", utilerrors.NewAggregate(newErrs)) </ins> } return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubectl drain crashed with panic when apiserver connection disappeared while there were pods that couldn't be evicted/deleted (sorry, the problem is a bit hard to reproduce and the log was lost) What you expected to happen: kubectl drain should exit with a proper error message How to reproduce it (as minimally and precisely as possible): Try kubectl drain multiple times, interrupting the connection during eviction/deletion Anything else we need to know?: The here can be in case if returns an error, and will panic upon accessing : This causes the c Environment: Kubernetes version (use ): The problem is present as of but is also present in earlier versions (I have a fix for this bug)
/priority important-longterm /sig cli


Question: ## `filepath-securejoin` ## <del> [![Build Status](https://travis-ci.org/cyphar/filepath-securejoin.svg?branch=master)](https://travis-ci.org/cyphar/filepath-securejoin) </del> <ins> [![Build Status](https://github.com/cyphar/filepath-securejoin/actions/workflows/ci.yml/badge.svg)](https://github.com/cyphar/filepath-securejoin/actions/workflows/ci.yml) </ins> An implementation of `SecureJoin`, a [candidate for inclusion in the Go standard library][go#20126]. The purpose of this function is to be a "secure"

One of the dependencies of k8s, filepath-securejoin, has a following vulnerability discovered: It has been patched in version 0.2.4, however k8s is still using version 0.2.3. Potential fix: updating the dependency. A small PR will follow. /sig security
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig security
/kind bug /sig security


Question: By("creating a file in subpath") cmd := "touch /volume_mount/mypath/foo/test.log" <del> _, err = framework.RunHostCmd(pod.Namespace, pod.Name, cmd) </del> <ins> _, _, err = f.ExecShellInPodWithFullOutput(pod.Name, cmd) </ins> if err != nil { framework.Failf("expected to be able to write to subpath") } By("test for file in mounted path") cmd = "test -f /subpath_mount/test.log" <del> _, err = framework.RunHostCmd(pod.Namespace, pod.Name, cmd) </del> <ins> _, _, err = f.ExecShellInPodWithFullOutput(pod.Name, cmd) </ins> if err != nil { framework.Failf("expected to be able to verify file") }

Which jobs are failing: Tests which exec into containers and alter lifecycles Which test(s) are failing: All new tests introduced for api change which exec into containers New alpha subpathExpr tests [Feature:VolumeSubpathEnvExpansion][NodeAlphaFeature:VolumeSubpathEnvExpansion] Since when has it been failing: 27/February/2019 Testgrid link: https://k8s- Reason for failure: is only set in e2e tests and the binary is not available in node e2e tests Need to restructure and call the API rather than through the framework convenience function Anything else we need to know:
/sig node /sig storage /assign /cc


Question: go_library( name = "go_default_library", <del> srcs = ["selinux_stub.go"], </del> <ins> srcs = [ "selinux_linux.go", "xattrs.go", ], </ins> importmap = "k8s.io/kubernetes/vendor/github.com/opencontainers/selinux/go-selinux", importpath = "github.com/opencontainers/selinux/go-selinux", visibility = ["//visibility:public"], <ins> deps = select({ "@io_bazel_rules_go//go/platform:android": [ "//vendor/golang.org/x/sys/unix:go_default_library", ], "@io_bazel_rules_go//go/platform:linux": [ "//vendor/golang.org/x/sys/unix:go_default_library", ], "//conditions:default": [], }), </ins> ) filegroup(

What happened: After upgrading from kubelet 1.15 -1.16, volumes are no longer relabeled as containerfilet for selinux, causing most pods to fail. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Run a kubernetes cluster on 1.15.[0-3]. Create a pod with a secret, configmap, or with provisioned storage (ebs-csi-driver). Find that file on disk of the node, and observe it is labeled with the file type containerfilet and the pod is able to read it. Upgrade the kubelet to 1.16 Delete the pod, and observe it fail on launch. Find the file on disk and see it labeled tmpfst or unlabeledt for configmap/secret and provisioned volumes, respectively. Anything else we need to know?: Environment: Kubernetes version (use ): 1.15 / 1.16 Cloud provider or hardware configuration: AWS OS (e.g: ): RHEL 7.7 Kernel (e.g. ): 3.10.0-1062.1.1.el7.x86_64 Install tools: kubeadm Network plugin and version (if this is a network-related bug): N/A Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Is selinux enabled for docker? Can you paster output of : also
Also please paste YAML of one or more pods which are affected by this. Thank you!
Security Options - Perhaps of note here, we do not change the installed by docker, and confirms we don't make changes to that file. Docker version: Client and server blocks match. Pod yaml incoming
To clarify, do you want a runnable "minimum reproducible" , or is it more informational? If the latter, I have a pod that breaks and I'm working on the yaml for that, removing sensitive information, etc. If the former, I'll need to spend a few minutes getting a minimal reproduction with busybox or somesuch.
I need full pod's yaml that is not working. i.e - generated full YAML.

Hi, I'm sorry I haven't had a chance to follow up with this ticket. I will get some pod yaml attached early next week. Thank you for following up.
I think I'm seeing the same thing, to reproduce, use the following resource YAML: On k8s v1.15.5 (hyperkube) On k8s 1.16.2 (hyperkube) Are you in Enforcing mode () ?
Hey thanks much for following on with example YAML. We've been totally swamped with work so I haven't had time to address this issue. To answer your question: we were seeing all pods with storage (configmaps, secrets, persistent volumes) failing in Enforcing mode, and disabling Enforcement () allowed the same pods to successfully run, with no problem. There was not as much output in audit logs as we would have expected, so constructing custom policies to allow access was a bit of shooting in the dark. We were able to get configmaps and secrets to work passably with a custom policy allowing limited access to tmpfs labeled items, but we can't get away with allowing access to unlabeledt in our compliance environment. Thanks again for following up with more info and linking up this ticket, and let me know if I can provide any more helpful information. We've rolled back to 15.3 for the time being, so I'll be constructing knowledge somewhat from memory, but the upgrade was run recently, and I took reasonably detailed notes, so my info is be largely accurate.
cc
Can someone post pod's YAML after pod was created. I am looking for some generated fields in the Pod.
I finally had the chance today to dig a bit deeper why SELinux is broken in 1.16+ In the linked PR you can see the fix. See the details in there. If you want to give that a try, I have build a binaries here: A workaround I have used in the meantime was: My reproduction of the bug can be found there: If you run the packer build the pods fail because of the SELinux problem
nice work! looks like the patch we need is this one right?
if so, can you please propose a PR against master of kubernetes/kubernetes?
Reopening to track adding a test.
On second thought, spawned a separate issue to track adding test coverage. /close
Closing this issue.

<detailsIn response to : On second thought, spawned a separate issue to track adding test coverage. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
we have just updated our cluster to 1.16.5, and even 1.16.6-beta.0, but the secret volume is still labeled as tmpfst:
I see the same result on the test above on v1.17.2
I can confirm the release binaries are not build correctly I have opened this issue in the release repo: /reopen Workaround is building the binaries yourself using bazel and the release tag
Reopened this issue. <detailsIn response to : I can confirm the release binaries are not build correctly I have opened this issue in the release repo: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
can someone confirm that 1.16.7 / 1.17.3 binaries are built with the selinux tags in make targets?
No they haven't as haven't been merged. Feel free to ask on those PRs again, I have not been successful, convincing that this should be merged with at least some priority
I am able to run this repro and get the desired outcome on 1.17.4. However, I am trying to attach Cinder volume whose context doesn't automatically get relabeled as containerfilet when I attempt to attach volumes with both in-tree and Cinder CSI plugin. I have filed this bug in the cloud-provider-openstack repo but not sure if this is a Kubernetes problem or a cinder-csi-plugin problem:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with .

Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten /assign are looking at adding selinux to CI using kops
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: // The second state change (optional) is due to the CPU burst parameter, which HPA again responds to. // TODO The use of 3 states is arbitrary, we could eventually make this test handle "n" states once this test stabilizes. func (scaleTest *HPAScaleTest) run(name, kind string, rc *common.ResourceConsumer, f *framework.Framework) { <ins> const timeToWait = 15 * time.Minute </ins> rc = common.NewDynamicResourceConsumer(name, kind, int(scaleTest.initPods), int(scaleTest.totalInitialCPUUsage), 0, 0, scaleTest.perPodCPURequest, 200, f) defer rc.CleanUp() hpa := common.CreateCPUHorizontalPodAutoscaler(rc, scaleTest.targetCPUUtilizationPercent, scaleTest.minPods, scaleTest.maxPods) defer common.DeleteHorizontalPodAutoscaler(rc, hpa.Name) <del> rc.WaitForReplicas(int(scaleTest.firstScale)) </del> <ins> rc.WaitForReplicas(int(scaleTest.firstScale), timeToWait) </ins> if scaleTest.firstScaleStasis > 0 { rc.EnsureDesiredReplicas(int(scaleTest.firstScale), scaleTest.firstScaleStasis) } if scaleTest.cpuBurst > 0 && scaleTest.secondScale > 0 { rc.ConsumeCPU(scaleTest.cpuBurst) <del> rc.WaitForReplicas(int(scaleTest.secondScale)) </del> <ins> rc.WaitForReplicas(int(scaleTest.secondScale), timeToWait) </ins> } }

Currently the time it takes for HPA and CA to scale up after load increase is not measured at all, and this can lead to regressions that will be missed. This issue is about adding an e2e test that will prevent such regressions.
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Filip, can you reopen? I cannot do this myself, and this issue is clearly not closed yet.
Thanks Wojtek!


Question: continue } <del> glog.Infof("[cpumanager] reconcileState: updating container (pod: %s, container: %s, container id: %s, cpuset: "%v")", pod.Name, container.Name, containerID, cset) </del> <ins> glog.V(4).Infof("[cpumanager] reconcileState: updating container (pod: %s, container: %s, container id: %s, cpuset: "%v")", pod.Name, container.Name, containerID, cset) </ins> err = m.updateContainerCPUSet(containerID, cset) if err != nil { glog.Errorf("[cpumanager] reconcileState: failed to update container (pod: %s, container: %s, container id: %s, cpuset: "%v", error: %v)", pod.Name, container.Name, containerID, cset, err)

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: On 1.8+ kubelet with --cpu-manager-policy=static, there is a constant stream of logging every 10 seconds or so in the following formats: for every pod, and for every privileged pod where update would fail but looks expected. What you expected to happen: First set of logs should probably be suppressed or to the debug level. Second set of logs should probably be better at distinguish expected case vs. unexpected case before logging that as errors. How to reproduce it (as minimally and precisely as possible): Run kubelet with , and watch /var/log/messages or similar. Anything else we need to know?: Environment: Kubernetes version (use ): 1.8.1 Cloud provider or hardware configuration: baremetal/onprem OS (e.g. from /etc/os-release): CentOS 7.4.1708 Kernel (e.g. ): 4.13.8- Install tools: custom Others:
/sig node
/cc
Thanks for filing this bug. Do you plan to work on fixing it too, or should we find someone to pick it up?
I can take a look and see what I can or cannot do with this.
"updating container" log line should be easy to deal with, it is doing a logging without setting a level, and if there is a convention on what should be the level of this type of logging (and similar in this file), I can submit a PR to add .V(x). support for it. Any preference on how the logging level should be handled?
I can help review . For the second part, it might be related to we left for ourselves in the CPU manager code. We should fetch the current resource config and only update the field.
volunteered to look into the second half of this.
Great. Regarding the second half, the best I can find so far indicates that the error is the result of attempting to modifying when the cgroup already has children. I will dig more also just to confirm.
Here is what I found:
Any update on the second part? Is it also fixed? Thanks.
This hasn't been fixed AFAIK. This error is only related to kube-proxy containers.
has there this issue related with kube-proxy been a solution, or it been ignorable?


Question: Name: "should-be-removable-validating-webhook.k8s.io", Rules: []v1beta1.RuleWithOperations{{ Operations: []v1beta1.OperationType{v1beta1.Create}, <ins> // This will not match any real resources so this webhook should never be called. </ins> Rule: v1beta1.Rule{ <del> APIGroups:   []string{"*"}, APIVersions: []string{"*"}, Resources:   []string{"*"}, </del> <ins> APIGroups:   []string{""}, APIVersions: []string{"v1"}, Resources:   []string{"invalid"}, </ins> }, }}, ClientConfig: v1beta1.WebhookClientConfig{

This flaked 3/4 times in the last week. /priority failing-test /priority important-soon /sig api-machinery /kind flake
/assign
Pinging here for latest status on this. This test is now one of our top flakes failing pull-kubernetes-e2e-gce job 40 times.
Here's the latest test run results for this test to get more logs. https://k8s-
The error message is really similar to the one in I'm looking into if the fix for that will also be sufficient to fix this.
We may want to reopen this until the fix is shown to actually work /reopen
Thanks for taking care of this. I will monitor this test and ping you back here to close when appropriate.
No failure 2 days since the fix. the fix seemed to have worked. So please go ahead and close the issue. Thanks much.
/close


Question: "-m", "set", "--match-set", proxier.ipsetList[kubeExternalIPSet].Name, "dst,dst", "-j", "RETURN") proxier.filterRules.Write( "-A", string(kubeIPVSFilterChain), <ins> "-m", "set", "--match-set", proxier.ipsetList[kubeHealthCheckNodePortSet].Name, "dst", "-j", "RETURN") proxier.filterRules.Write( "-A", string(kubeIPVSFilterChain), </ins> "-m", "conntrack", "--ctstate", "NEW", "-m", "set", "--match-set", proxier.ipsetList[kubeIPVSSet].Name, "dst", "-j", "REJECT")

We are running self managed Kubernetes clusters on GCP and use Services of with both public and private s to create load balancers for our clusters. Since upgrading kube-proxy to 1.26.3 (from 1.25.5) we saw that all healthchecks from GCP started to fail and traffic is now blindly load balanced across all nodes, which broke Services with set to . The issue seems to be introduced with this change: Here, ipset is introduced and is used in iptables on chain as: seems to include external IPs (as also noted here: and here: ) so the last rule above is dropping packets to external IPs that are not routed before reaching this point. The next chain in iptables is and contains a single rule: which seems specific to route packets arriving to health check ports Workarounds to get health checks to work again from GCP are to either flush ipset (temporary since kube-proxy will fight back) or move the ule to route packets to health check packets on top of : Health checks from GCP should continue to work as intended after upgrading kube-proxy from 1.25.5 to 1.26.3 Use kube-proxy v1.26.3 in a Kubernetes cluster on GCP and create a Service of type . Health checks from GCP should fail on all nodes Rolling back to v1.25.5 seems to fix the issue, even though we have to roll to fresh nodes to clean up everything and make health checks work again. <details</details<detailsGCP </details<details</detailsNo response <detailscontainerd://1.6.8 </details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig network
/cc /area ipvs
/assign
I can't recreate this problem on K8s v1.26.4. I use a simple service: And get: The wget is from an external node on the same segment.

192.168.1.2 is the address to a node where a server exist, 192.168.1.4 to a node without a server. Is this happening in combination with I.e. are you using a node-ip as loadBalancerIP?
Hi I have the same problem. A packet capture reveals, in GCP the healthcheck is directed to nodes with the LB IP as destination ip. So in order to create this problem either create a static arp entry for the loadBalancer IP, and wget http://loadBalancerIP: or spinup a pod in the cluster and do a http://loadBalancerIP:
Yeah, this is related to my comment here , the loadbalancer IP is as a local route to the host
Thanks, I have treated them as separate issues. I will check again.
yes, as pointed out, in order to reproduce this you need to make requests to the LB address. I can also curl kube-proxy on localhost fine, but gcp goes via the LB addresses to health check
To use http://loadbalancerip:/ doesn't make sense to me unless you have a one-node cluster with loadbalancerip==nodeip. The probe is to check if an endpoint is present on a particular node, hence the loadbalancerip can't really be used. But the comment: indicates that you have many nodes. I don't get it.
It appears that GCE ILBs rely on Direct Server Return (DSR) as their load balancing mechanism. DSR only modifies the destination MAC address and is utilized during health checks across all VMs within the associated instance groups. This is likely due to K8S being just one of many use cases, and health checks may be conducted on the same port as the load-balanced service. In such cases, the destination IP serves as the distinguishing factor on the host.
yes, I have many nodes on my clusters. Catching health check traffic on a node for the following service as an example: gives: The controller configures a health check with 8 seconds interval on port , which is also visible from the packets arriving on the node. I am not 100% on how GCP routes for those health checks but feels like comment above is spot on.

So to summarise; GCP sends probes to "loadbalancerip:health-check-port" in intervals and explicitly set the mac address to target different nodes. Not the cleanest solution imho, and very hard to imagine beforehand. Here is what I suggest, add a rule before the reject in KUBE-IPVS-IPS:
yes, this is a workaround that seems to do the trick (I also mention that one in the issue description). Currently, kube-proxy already adds such a rule, but this happens in chain which is evaluated after . I think it is a matter of ordering rules
I don't want to reorder, because some other corner-case might pop up. But to add that rule, and keep everything else as-is feels safer. It will however expose the healthCheckPort externally via the loadBalancerIP. In GCP I am sure that is prevented by some fw-rule somewhere, but other K8s installation may be affected.
Many thanks for your explanations
I have created a , but I can't test that it really solves the problem on GCP. Can any of you please apply the PR and verify that it works on GCP?
Would it be possible to configure GCP to use the same port as the load-balanced service also for K8s? I.e. don't use the healthCheckPort at all. A problem may be that a request to a node without a server may be black-holed, i.e. no reject either icmp or http.
this is more complex since is how cloud-providers loadbalancers implemented it, using externalTraficPolicy: local, the health check is on the , for externalTraficPolicy: cluster, some cloud-providers use the applications port and others use the kube-proxy healtcheck, that it runs on any of the node ips
Imo key things are: and: So from GCP perspective they are not aware of what's running on the cluster. K8s or something else. So the probing for available servers are generic, "K8S being just one of many use cases". In that case I assumed that the cluster admin may configure how probing is done.

But I don't know GCP so it may not be so, but... On the other hand if GCP is aware that this is actually a K8s system, the "K8S being just one of many use cases" doesn't hold and GCP should probe the node-ip's instead of using some MAC-hack.
I can confirm that I have compiled the PR, updated the kube-proxy ds image, and tested it on GCP. It is working as expected! :fireworks: You're welcome! I'm glad I could be of help. Your Metallb patches have also been very useful to me in the past. Thank you for fixing this issue! this is more complex since is how cloud-providers loadbalancers implemented it Although I could patch the cloud-provider-gcp controller for my own use, this wouldn't solve the issue for other users or other cloud controllers. Additionally, using the service port itself may result in hitting the application directly, which may not always be desired.


Question: -o -path './test/e2e/*'  -o -path './test/e2e_node/*'  -o -path './test/integration/*'  <ins> -o -path './test/component/scheduler/perf/*'  </ins> ) -prune  ) -name '*_test.go' -print0 | xargs -0n1 dirname | sed 's|^./||' | sort -u )

We have found that on 1k nodes cluster scheduling takes 60-200 ms and becomes the bottleneck. We want to have a standard way to reproduce scheduling latency metrics result and benchmark scheduler as simple and fast as possible. Only with that can we: 1. figure out underlying behaviors and potential bottlenecks quickly; 2. publish results in a standard way and prevent future performance degradation. We have the following goals: Goal 1: Save time on testing. More accurate scheduling latency. We mock up kubelets and only measure scheduling latency and do not care about the latency to start the pod. This is different from Kubemark -- scheduled pods won’t be run by mocked kubelets. We only need to set up the master components. Goal 2: Profiling runtime metrics and find out performance bottleneck. Write scheduler integration test. Take advantage of go test and profiling tools, like cpu-profiling, memory-profiling and block-profiling. This is very fine-grained metrics. Goal 3: Reproduce test result easily. We want to have a known place to do the performance related test for scheduler. Developers should just run one script to collect all the information they need. We have the following plans for writing scheduler tests: density test (by adding a new Go test) schedule 30k pods on 1000 (fake) nodes and 3k pods on 100 (fake) nodes print out scheduling rate every second let you learn the rate changes vs number of scheduled pods benchmark (by modifying the existing benchmark in integration or simply add a new one) make use of and report nanosecond/op. schedule b.N pods when the cluster has N nodes and P scheduled pods. Since it takes relatively long time to finish one round, b.N is small -- level of 10. We are new to k8s codebase. We also want to know where is the best place (directory) to commit in these tests? Any suggestions are welcomed!
/cc
This extends
So density already does this... the one option it looks like you may want is custom control , real or fake is irrelevant as the e2e's just need to point to an api-server. re: benchmarks, seems like we could simply add to the unit-tests.
After chatting with lets add this as an agenda item for the SIG meeting tomorrow.

+1 for adding benchmark (howeveer this should be integration test, not unit-test in my opinion)
cc The next SIG on scheduler meeting is on 12.07, isn't it? If there is one tomorrow, can I have the information please?
retract, +1 to . I was adding this to the list given the context, but it crosses over into both.
Slides from today's meeting:
Here are are the action item notes from the SIG meeting: Everyone agrees this is a good idea and long overdue, but the question is where would this live: New category of tests? (isolated/sandbox’d/component) The current testing structure allows to easily append. create new directory (component) + hack/script to execute Future "we could" gate on known good data, and fail performance regressions
/cc
/cc


Question: PodTerminated PodStatus = "Terminated" // PodUnknown means that we failed to obtain info about the pod. PodUnknown PodStatus = "Unknown" <ins> // PodSucceeded means that the pod has stopped without error(s) PodSucceeded PodStatus = "Succeeded" </ins> ) type ContainerStateWaiting struct {

This happened to me several times while debugging an e2e test. My branch was based on commit and had changes only to
docker inspect says the container exited 0
kubelet HTTP says the pod exited 0:
kubectl says it failed:
Events does not show any success or failure even. Seems like it doesn't show that for any pods, succeeded or failed.
Apiserver sees the exit code 0:
The problem might be related to there being zero values for startedAt or finishedAt. The command may have exited so quickly that docker could not record these values, and we may be somehow treating unset values for these as failure.
adding sleep 5 to my command causes docker to set the startedAt and finishedAt to reasonable values, but the pod still shows up as failed. Note that I use a different pod name each time I try to avoid that sort of obvious mistake.
So, the apiserver v1beta1 message knows that it is terminated with exit code 0. But, both kubectl and my go code in my go e2e test thing this means failed. I'm starting to suspect the conversion code.
The problem is that a v1beta1 pod can never see that a pod is failed or succeded. So it defaults to PodFailed. Grrrrrrraaaaahhhhhh!
In line 184 of
This is going to be confusing for people unless we change this.
I believe the latest fix for PodStatus should resolve the issue here. I will double confirm it before closing it.

We should do this after 1.7. xref /sig testing cc
Is this the task that needs completed - Add support to Ubuntu e2e test-infra
That is different, this is about running GPU tests as part of PR pre-submit tests. Right now we skip them because they are "[Serial]".
I can work on this issue in conjunction with . Is this issue just a matter of adding GPU tests to a different suite or are there more subtasks?
Anymore progress on this?
Thanks for the reminder. I just submitted an internal request for quota to run these tests. Once that is approved, I will add a PR in test-infra to run GPU tests on all k/k PRs. It will be non-blocking initially and then become blocking later on.
Awesome, thanks!


Question: "k8s.io/client-go/util/workqueue/default_rate_limiters_test.go", "k8s.io/client-go/util/workqueue/delaying_queue_test.go", "k8s.io/client-go/util/workqueue/rate_limitting_queue_test.go", <del> "k8s.io/client-go/util/workqueue/timed_queue_test.go", </del> ], library = ":k8s.io/client-go/util/workqueue", tags = ["automanaged"], deps = [ "//vendor:k8s.io/apimachinery/pkg/util/wait", <del> "//vendor:k8s.io/client-go/pkg/api/v1", </del> "//vendor:k8s.io/client-go/util/clock", ], )

The TimedWorkQueue object appears to lack a critical feature of the original WorkOueue, namely that similar items are aggregated together. By adding the time member to to the key type, it ensures that (time, item) pairs are unique-- you can add the same item multiple times as long as the time differs. This removes the guarantee that two goroutines wouldn't process the same item at the same time.
Iirc, the timed queue feeds a regular queue, so the guarantee still holds
Sorry, I was thinking of the delaying queue, which does key correctly
I'm going to remove timed workqueue. It's used to gather metrics for queues used in the garbage collector. This feature is supported by a regular workqueue now.


Question: ValidateOrFail(k8s, model, &TestCase{ToPort: 80, Protocol: v1.ProtocolTCP, Reachability: reachability}) }) <ins> ginkgo.It("should enforce ingress policy allowing any port traffic to a server on a specific protocol [Feature:NetworkPolicy] [Feature:UDP]", func() { nsX, _, _, model, k8s := getK8SModel(f) policy := GetAllowIngressByProtocol("allow-ingress-by-proto", map[string]string{"pod": "a"}, &protocolTCP) CreatePolicy(k8s, policy, nsX) reachabilityTCP := NewReachability(model.AllPods(), true) ValidateOrFail(k8s, model, &TestCase{ToPort: 80, Protocol: v1.ProtocolTCP, Reachability: reachabilityTCP}) reachabilityUDP := NewReachability(model.AllPods(), true) reachabilityUDP.ExpectPeer(&Peer{}, &Peer{Namespace: nsX, Pod: "a"}, false) ValidateOrFail(k8s, model, &TestCase{ToPort: 80, Protocol: v1.ProtocolUDP, Reachability: reachabilityUDP}) }) </ins> ginkgo.It("should enforce multiple ingress policies with ingress allow-all policy taking precedence [Feature:NetworkPolicy]", func() { nsX, _, _, model, k8s := getK8SModel(f) policyAllowOnlyPort80 := GetAllowIngressByPort("allow-ingress-port-80", &intstr.IntOrString{Type: intstr.Int, IntVal: 80})

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Add missing unit test cases based on these and fair deviations when applied. - Allow Ingress traffic to an application</strike- - DENY all traffic from other namespaces (w/o NS selector)</strike- - Allow traffic from other namespaces (w/o NS selector) - Allow traffic multiple selector (OR) - DENY default-deny-egress from an app</strike- - DENY default-deny-all-egress from a namespace</strike- </strikeWhy is this needed: Increase the coverage of netpol tests Ref
/area tests /sig network
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /area tests /sig network Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
- Map of current netpol created by each test and it's descriptions.
hello can i take this issue?
feel free to come up with the ones in the list, just raise here the one you'll get and I update the comment.
maybe i take number 4
great, we are aiming at the new netpol e2e framework, some other examples:
/triage accepted /assign Amin, do you think this is a good first issue, with the current framework? If so we can gather some new contributors, and use your and skills for the review of the PRs :D
Can we carve this into 5 separate PRs ? I know was planning on taking a couple , but not sure which ones he was doing
YEap, this is the idea :D 1 issue, 5 or more PRs, good first issue, each people that wants pick one :)
absolutely, I think it applies since we have a lot of examples already /good-first-issue
2 other checkboxes plz :) can you add a checkbox for windows tests as well ? can you also add a checkbox for UDP coverage increases as well ? Right now we need to make sure some subset, at least , of the netpol suite will work on windows nodes, and curate the tags accordingly. Im investigating this on TKG (antrea) and AKS (calico) at the moment, and will collab w/ on the details ..
sure.

LMK if you guys need more granularity on the list
Can I take up 5 - Allow traffic from other namespaces (w/o NS selector) ?
sure, LMK if you have any questions
great sounds like this is all carved up :) thanks
important note: <-- can you also please make sure your test cases arent already covered ? We have about 31 test cases in the new framework now. So, you might only need to verify that the tests exist/add a comment to them to describe them properly, in some cases. ( Lets try not to add redundant tests )
we are good to go. /remove-good-first-issue
/remove-help
- Getting the diff between the new and old suites coverage. one has one more test - one has UDP - The new ones under have more tests. This is true for . We are missing coverage of:
any progress here? I'm available on Slack if you guys want to chat.
/close Windows tests are on
Closing this issue. <detailsIn response to : /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Using this issue as umbrela to:
Can we add the endPort test here, just to increase this coverage?
moved the port range tests to


Question: func CheckDaemonPodOnNodes(f *framework.Framework, ds *appsv1.DaemonSet, nodeNames []string) func(ctx context.Context) (bool, error) { return func(ctx context.Context) (bool, error) { return checkDaemonPodStateOnNodes(ctx, f.ClientSet, ds, f.Namespace.Name, nodeNames, func(pod *v1.Pod) bool { <del> return podutil.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) </del> <ins> return podutils.IsPodAvailable(pod, ds.Spec.MinReadySeconds, metav1.Now()) </ins> }) } }

Several test/e2e/framework/* packages have as their only dependency on k/k. The following functions are shared that way: IsPodAvailable returns true if a pod is available; false otherwise. IsPodReady returns true if a pod is ready; false otherwise. VisitContainers invokes the visitor function with a pointer to every container spec in the given pod spec with type set in mask. If visitor returns false, visiting is short-circuited. VisitContainers returns true if visiting completes, false if visiting was short-circuited. AllFeatureEnabledContainers returns a ContainerType mask which includes all container types except for the ones guarded by feature gate. The feature gate check in AllFeatureEnabledContainers is problematic. E2E tests don't know what the feature gates in the cluster are. The rest of the functions look like something that could be moved into == Having the functions in makes it clearer that the checks only depend on aspects of the public API. Reduces dependencies of e2e/framework on k/k, in case that we want to move it into staging. Prevents accidentally using checks that only work in-tree (like feature gates).
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/assign


When we want to rotate certificates for aggregated API servers, we have to restart (kill) pods in order for API server to pick the new certificates. It would be very useful to implement some sort of self-awareness for the aggregated API servers, so they can pick-up new certificates when they're updated, without user interaction. If we have certificate files in pods, that are provided by some volumeSource, that can be persistent storage or a Secret, when file gets changed on the volume, the kubelet updates file in the pod usually in matters of few seconds. However, aggregated API servers are not self-aware of such change and users have to manually restart the pods. This brings additional complexity, especially if you want to automate the process. The first problem is that you need to know what pods you need to restart. But the potentially biggest problem is that you need additional permissions to do so. That would include RBAC permissions for listing pods, listing deployments and for deleting pods. That can make a security issue in some setups. By making aggregated API servers sort of self-aware, this could be avoided. Create a health check somewhere in , that utilizes a file watcher to watch for certificates changes. When certificate is changed, fail a health check. Failing health check will restart the pod without user interaction. This could be the most easiest and simplest solution to the problem. Such feature should/could be optional and guarded by a flag, let's name it . Some delay could be required between restarting pods. We probably don't want to kill all pods in the same time. This is up to be discussed. I would like to work on this issue, so feel free to assign it to me. /kind feature /sig api-machinery /area apiserver /cc
this could also be done by a purpose-built container that watched the same input files. as you point out, you likely would not want to restart all apiservers at once, which makes it unclear whether you'd want to build in the restart logic to the individual servers
That coordination would certainly work and be a valuable tool for someone to build to help manage rolling generic processes if we never get a pod restart. However, being able to have a "usually won't all go down together" option built-in is easy to understand and will resolve the problem until then.

It isn't a stretch to say that if your process isn't fulfilling the current intent of the pod, it isn't healthy. This gives us a thing between dynamic config reloading and nothing. A container that can coordinate based on a service would be really cool, but I think that'd be a separate thing.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details