Question: <ins> /* Copyright 2014 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package util import ( "hash" "github.com/davecgh/go-spew/spew" ) // DeepHashObject writes specified object to hash using the spew library // which follows pointers and prints actual values of the nested objects // ensuring the hash does not change when a pointer changes. func DeepHashObject(hasher hash.Hash, objectToWrite interface{}) { spew.Fprintf(hasher, "%#v", objectToWrite) } </ins>

Is this to follow pointers?
Context: The follows pointers when printing. Unlike JSON marshaling/unmarshaling it also handles private fields.

I am experiencing some weird issues when enabling HTTP liveness probes for some containers. I configure the following for all HTTP-serving containers (this is part of a template used by a replication controller with a single replica): When this is configured, the following issues start to occur: One pod is already running without problems. When I add another replication controller which creates a new pod (also with the liveness probe set), the first pod (the one that was already running) is restarted. This happens even when the two pods are in different namespaces. But when a pod's HTTP server fails completely (connection to the port is refused), the liveness probe will not trigger and the pod will not be restarted. This behavior occurs consistently when I start with no replication controllers and then add them one by one. When I disable the liveness probes on the affected pods/templates, they are no longer restarted in this manner.
You say you're starting two replication controllers-- can you verify that their label selectors do not intersect?
Yes, each replication controller has a selector over a single label in the form of so the labels do not overlap. Also, this problem occurs even when the apps are in different namespaces and goes away when the liveness probe is removed.
Thanks-- last question-- Do the pods have to be scheduled on the same node for this to occur? Your second point I believe is a problem here: we treat an error during HTTP GET as healthy for some inexplicable reason. I can't really explain your first observation, unless perhaps the pods and containers have the same names, but in different namespaces AND kubelet has a namespace handling bug?
cc
"Unknown" != "Healthy" I think it's an open question about whether we want to treat a GET failure as "Unhealthy" we certainly could... --brendan , Daniel Smith wrote:
Giving this to Saad. I think it's pretty clear that if kubelet can't do an HTTP GET, then the pod isn't healthy-- they're on the same host, HTTP get can't fail for network reasons. Our internal system makes this assumption. (I broke it once!)
tl;dr... When a second replication controller is the hash of the container of the first replication controller changes causing it to be killed and recreated. The hash changes because the LivenessProbe pointer changes. Need to figure out why that happens.

Spent time trying to understand why creating two replication controllers, both with http liveness probes, causes the container from the first replication controller to be killed and then recreated when the second replication controller is created. It turns out, as soon as the second replication controller is created, the hash of the container from the first replication controller changes and thus the container is killed: Let's see how the expected and actual hashes are computed and how they end up differing. The actual hash is pulled from the docker container name (dockerContainers.FindPodContainer): The expected hash is created from the object. Specifically, this code (): Expected hash before second replication controller is : Expected hash after second replication controller is : The difference between the container before and after is: So the hash changes because the LivenessProbe pointer in the object changes. Need to figure out why this happens.
Great detective work-- I think it's clear that we shouldn't be including pointers in our hash, so we need to revisit that and explicitly hash only the things that we care about.
Adding second replication controller changes the pointer of LivenessProbe in container config is another mystery. We need to make sure not randomly update other fields too.
Dug in to this some more: confirmed that the value of the original container's LivenessProbe doesn't actually change when the second replication controller is created--only the pointer to the LivenessProbe object changes. It turns out that the instance of the  object being used to compute the expected hash is not guaranteed to be the same instance for the lifespan of the actual docker container. In fact, every time there is an etcd update (e.g. a new pod is created) a new boundpods object is created and although the values of the existing pod/container container objects don't change, because new instances are created the pointers do change (by design). So the fix for this is to ensure that the expected hash is computed only on container values not pointers. One way to ensure this would be to make sure the container object specifies no pointer fields. However, LivenessProbe and Lifecycle, since they are optional, must be able to be set to . Thus, pointer fields can't simply be converted to value fields. Another approach would be to only hash some subset of the values (maybe the name and image) of the container.

However, containers should be destroyed/recreated (meaning the hash must change) if the actual values of any of the container fields changes (e.g. the cpu, memory, volume mounts, liveness probe path, etc.), not just the container name/image. Thus, pointer fields can't simply be disregarded for the hash computation. Instead the fix must modify the expected hash to be computed on the complete deep values of the entire container object (including the actual LivenessProbe values, instead of the pointer). The easiest way to do this is to marshal the container object to json and compute the hash from the resulting byte array.


Question: cp -a "${SERVER_BINARY_TAR}" ${local_dir} cp -a "${SALT_TAR}" ${local_dir} <del> aws s3 sync --exact-timestamps ${local_dir} "s3://${AWS_S3_BUCKET}/${staging_path}/" </del> <ins> aws s3 sync --region ${s3_bucket_location} --exact-timestamps ${local_dir} "s3://${AWS_S3_BUCKET}/${staging_path}/" </ins> <del> aws s3api put-object-acl --bucket ${AWS_S3_BUCKET} --key "${server_binary_path}" --grant-read 'uri="http://acs.amazonaws.com/groups/global/AllUsers"' </del> <ins> aws s3api put-object-acl --region ${s3_bucket_location} --bucket ${AWS_S3_BUCKET} --key "${server_binary_path}" --grant-read 'uri="http://acs.amazonaws.com/groups/global/AllUsers"' </ins> SERVER_BINARY_TAR_URL="${s3_url_base}/${AWS_S3_BUCKET}/${server_binary_path}" local salt_tar_path="${staging_path}/${SALT_TAR##*/}" <del> aws s3api put-object-acl --bucket ${AWS_S3_BUCKET} --key "${salt_tar_path}" --grant-read 'uri="http://acs.amazonaws.com/groups/global/AllUsers"' </del> <ins> aws s3api put-object-acl --region ${s3_bucket_location} --bucket ${AWS_S3_BUCKET} --key "${salt_tar_path}" --grant-read 'uri="http://acs.amazonaws.com/groups/global/AllUsers"' </ins> SALT_TAR_URL="${s3_url_base}/${AWS_S3_BUCKET}/${salt_tar_path}" }

When I run: I get the following error: Waiting a bit and running got past this point because the bucket was ready the second time around, but the getting-started process should be more robust.
I experienced the same issue
We do try to query for the bucket existing, but perhaps S3 is even more eventually consistent than I thought! I did observe that we were very trusting about the user providing the correct S3 region, and have tried to fix this in . I don't think that was the problem here though (if you make_bucket, you can't have a region mismatch...) I guess we could retry the sync a few times; might help with e.g. a network interruption during the sync also.
I believe this to have been fixed by . I'm going to close, but if it happens again please do reopen.
FYI I got this with the latest k8s as of 10/13/15: I believe this is k8s 1.0.6 but I just tore down my AWS cluster and apparently kubectl needs to talk to the now-terminated master instance. I ran to get k8s. Waiting and running again doesn't show the issue since the bucket is fully created and accessible. I just provisioned a Vagrant cluster in hopes that would help show what version I'm using. Here's the output of :


Question: if file == o.OldDriverName { *p = path.Join(dir, o.NewDriverName) } <ins> // Inject non-standard kubelet path. *p = substKubeletRootDir(*p) </ins> } } }

What happened: The CSI hostpath driver used in the e2e tests expects the root directory of nodes to be set to , , however nodes can be configured to use different paths. The tests should find out the root directory from kubelet to allow for this. Several test failures related to CSI Volumes: What you expected to happen: Test should be expected to pass. How to reproduce it (as minimally and precisely as possible): Configure kubelet on cluster nodes to use the non-default path. Run the storage-csi e2e tests. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig testing
/sig storage /remove-sig testing This is a SIG storage specific issue.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
How? I've asked about that before and the conclusion was that there is no portable way to do that (i.e. no official API). It seems to fall onto cluster admins to configure CSI driver deployments so that this particular aspect matches the cluster. For the tests that means that there has to be a parameter for the kubelet data directory that users of then have to set.
We could add and respect a flag of course then all nodes have to be the same at least. Reminder though
That's indeed another complication.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale Tracked for csi-driver-host-path in There's a bigger issue about both mock and hostpath driver in which includes "support non-default kubelet path".
The current hostpath driver in k/k testing supports non-default paths, but the suite doesn't use that yet.

/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: // ReplicationController represents the configuration of a replication controller. type ReplicationController struct { <del> TypeMeta   `json:",inline"` </del> <ins> TypeMeta `json:",inline"` // If the Labels of a ReplicationController are empty, they are defaulted to be the same as the Pod(s) that the replication controller manages. </ins> ObjectMeta `json:"metadata,omitempty" description:"standard object metadata; see https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/api-conventions.md#metadata"` // Spec defines the desired behavior of this replication controller.

It doesn't make sense to define pod template with a set of labels that is strict subset of replicaSelector (or at least I can't find an use-case). For that reason I think that the labels in pod template part of replication controller should be optional and user should care about them only if he wants to define a superset of what replicaSelector already defines.
I've heard the same comments from a client. I think that it'd be more intuitive if we always specify pod labels in the template (so that we can copy&paste pod definition), and replicaSelector is optional (by default pod labels).
It would be reasonable for the selector to be a subset. I'm fine with intelligent defaulting but we need to be careful not to conflate the 2 concepts, which I think would increase the amount of confusion.
Setting RC selector from pod template labels SGTM. Any concerns It wouldn't have to work with separate templates, though it would be desirable if it did. It would be nice if we could add preconditions based on other objects (resource version of pod template in this case) to etcd transactions.
I think we agreed in another issue that defaulting the selector from the pod template labels is reasonable. No argument from me. One problem with pod templates is that we lose the strict relationship between selector and template, so if you change the template the RC needs to behave intelligently (I refuse to create more pods because the template selector doesn't match mine, etc)
We should do this, and we should default replicas to 1. We could also consider defaulting replication controller labels from pod template labels. That way, essentially the only thing that would need to be specified for a singleton RC would be the pod template.
I don't think we can default replicas to 1, since we can't tell the difference between an explicit zero and a default zero, and I think there are lots of reasonable cases where you want to set replicas == 0 on create. (esp. in the context of socket activation)
Responded on the PR: We can distinguish default from 0 if we make it a pointer.


Question: 1 /* expectedSetUpCallCount */, testKubelet.volumePlugin)) // Remove pod <del> kubelet.podWorkers.(*fakePodWorkers).removeRuntime = map[types.UID]bool{pod.UID: true} </del> <ins> kubelet.podWorkers.(*fakePodWorkers).setPodRuntimeBeRemoved(pod.UID) </ins> kubelet.podManager.SetPods([]*v1.Pod{}) assert.NoError(t, waitForVolumeUnmount(kubelet.volumeManager, pod))

pull-kubernetes-unit TestVolumeUnmountAndDetachControllerDisabled It failed on but after that the test is passing DATA RACE It seems that the data race is caused by below code It failing of test is Flaky, it was not reproducible on my local machine too, I have used below command to run the test output on my local machine I would like to work on this issue, is it possible to reproduce this issue locally?
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
SIG Testing owns test frameworks, CI infrastructure, and other things to support testing. But this test is owned by SIG storage I believe since it tests their code, as we own tests that test our code.


Question: } }) <ins> // quick validation of udp, next test confirms that this services update as well after endpoints are removed, but is slower. ginkgo.It("should support basic nodePort: udp functionality", func() { config := e2enetwork.NewNetworkingTestConfig(f, true, false) ginkgo.By(fmt.Sprintf("dialing(udp) %v (node) --> %v:%v (nodeIP) and getting ALL host endpoints", config.NodeIP, config.NodeIP, config.NodeUDPPort)) err := config.DialFromNode("udp", config.NodeIP, config.NodeUDPPort, config.MaxTries, 0, config.EndpointHostnames()) if err != nil { framework.Failf("Failure validating that nodePort service WAS forwarding properly: %v", err) } }) </ins> // Slow because we confirm that the nodePort doesn't serve traffic, which requires a period of polling. ginkgo.It("should update nodePort: udp [Slow]", func() { config := e2enetwork.NewNetworkingTestConfig(f, true, false)

What happened: In CAPA clusters, NodePorts are disabled for UDP but not for TCP, so when testing, we see a small hole in coverage that was luckily caught by the tests.... So, I noticed that we thus don't have a targetted ginkgo test for UDP + nodeports, but we do seem to have the Contract one, which I guess tests two things at once: UDP NodePort service works Conntrack can persist the service We do have but this test is obviously testing many other things as well. So, would like to have a test.... What you expected to happen: running the sig-network test suite, while skipping with UDP not working would result in a test failure specific to UDP and NodePorts. How to reproduce it (as minimally and precisely as possible): Run the test suite, skip , with a UDP firewall up - you'll see no relevant failures. Anything else we need to know?: Environment: Kubernetes version (use ): master
/sig network
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
There is already one testing NodePort UDP in
However, the more tests the better :+1:
Hiya Antonio. I think the one your talking about in granular checks is also tagged as "Slow"..... Is there a FAST one? If so I'll close this issue don't want to make a redundant test . :)
I honestly don't know the history of the Slow tag and I don't see it consistently applied, but if you think that you can improve the Granular check and have an additional faster test, then just go for it :muscle:
ok cool


Question: CLEANUP=true GINKGO_FLAGS='--skip="[Flaky]|[Slow]|[Serial]"' SETUP_NODE=false <del> TEST_ARGS=--cgroups-per-qos=true </del> <ins> TEST_ARGS=--cgroups-per-qos=false </ins>

In node-e2e-test on Containervm image, the containers of the pods created by Kubernetes are subcontainers of docker-daemon/docker/. It means that the runtime resource usage (docker-daemon) is not correct, because all pods are included. This problem maybe due to container misconfiguration in node-e2e-test.
The kubelet on debian-based containervm image is misconfigured. Will fix it shortly.


Question: svc.Spec.LoadBalancerSourceRanges = []string{acceptPod.Status.PodIP + "/32"} }) <del> // Clean up loadbalancer service </del> defer func() { <del> jig.UpdateServiceOrFail(svc.Namespace, svc.Name, func(svc *v1.Service) { svc.Spec.Type = v1.ServiceTypeNodePort svc.Spec.LoadBalancerSourceRanges = nil }) err := cs.CoreV1().Services(svc.Namespace).Delete(svc.Name, nil) framework.ExpectNoError(err) </del> <ins> ginkgo.By("Clean up loadbalancer service") e2eservice.WaitForServiceDeletedWithFinalizer(cs, svc.Namespace, svc.Name) </ins> }() svc = jig.WaitForLoadBalancerOrFail(namespace, serviceName, loadBalancerCreateTimeout)

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gce-scale-correctness Which test(s) are failing: [sig-network] Services should only allow access from service loadbalancer source ranges [Slow] Since when has it been failing: It has been failing since 8/28. Here is the k/k diff for additional context: Testgrid link: Reason for failure: Seems like they were two instances of the same test (at least in the latest run). One of them failed with the following error message: The other one with: /milestone v1.16 /priority critical-urgent /kind failing-test /cc /cc /cc
/sig network
- are you aware of any recent changes that might affect this tests (e.g. judging just by title)? It fails deterministically.
/remove-kind bug Actually
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Those labels are not set on the issue: <detailsIn response to : /remove-kind node /assign - any chance this is related to ? This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted () Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /remove-sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Those labels are not set on the issue: <detailsIn response to : /remove-kind node /assign - any chance this is related to ? This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted () Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Any chance this is related to ?

This is a little wild guess, but from logs it seems that the namespace wasn't deleted, because objects other that pods were not deleted ()
can you please let the 1.16 release team if you are going to be able to investigate and provide if you can help an approximate time for a fix?
Acked, I'm looking into the failure and will provide an update soon.
Fantastic! Thanks for the update
Yep, this is related to With service finalizer enabled by default, deleting the namespace (including services) now takes longer --- it now blocks on all the underlying load balancer resources to be cleaned up. It is expected that the latency for resource clean up might be longer on large cluster (such as in this test). I will soon send a PR to overwrite the  with something more reasonable in this case.
Sent to fix this.
Reopening this one to track the job's health /reopen
Reopened this issue. <detailsIn response to : Reopening this one to track the job's health /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yesterday's run has failed due to problems with creating a test cluster. Hopefully we get an answer whether the fix has helped today.
Thanks for tracking this, seems like the latest run is green:
It looks good indeed, thanks /close
Closing this issue. <detailsIn response to : It looks good indeed, thanks </details


Question: }) } } <ins> func TestPodVolumesExistWithMount(t *testing.T) { poduid := types.UID("poduid") testCases := map[string]struct { prepareFunc func(kubelet *Kubelet) error expected    bool }{ "noncsivolume-dir-not-exist": { prepareFunc: func(kubelet *Kubelet) error { return nil }, expected: false, }, "noncsivolume-dir-exist-noplugins": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) return os.MkdirAll(filepath.Join(podDir, "volumes/"), 0750) }, expected: false, }, "noncsivolume-dir-exist-nomount": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) return os.MkdirAll(filepath.Join(podDir, "volumes/plugin/name"), 0750) }, expected: false, }, "noncsivolume-dir-exist-with-mount": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/plugin/name") if err := os.MkdirAll(volumePath, 0750); err != nil { return err } fm := mount.NewFakeMounter( []mount.MountPoint{ {Device: "/dev/sdb", Path: volumePath}, }) kubelet.mounter = fm return nil }, expected: true, }, "noncsivolume-dir-exist-nomount-withcsimountpath": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/plugin/name/mount") if err := os.MkdirAll(volumePath, 0750); err != nil { return err } fm := mount.NewFakeMounter( []mount.MountPoint{ {Device: "/dev/sdb", Path: volumePath}, }) kubelet.mounter = fm return nil }, expected: false, }, "csivolume-dir-exist-nomount": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/kubernetes.io~csi/name") return os.MkdirAll(volumePath, 0750) }, expected: false, }, "csivolume-dir-exist-mount-nocsimountpath": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/kubernetes.io~csi/name/mount") return os.MkdirAll(volumePath, 0750) }, expected: false, }, "csivolume-dir-exist-withcsimountpath": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir(poduid) volumePath := filepath.Join(podDir, "volumes/kubernetes.io~csi/name/mount") if err := os.MkdirAll(volumePath, 0750); err != nil { return err } fm := mount.NewFakeMounter( []mount.MountPoint{ {Device: "/dev/sdb", Path: volumePath}, }) kubelet.mounter = fm return nil }, expected: true, }, } for name, tc := range testCases { t.Run(name, func(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet if tc.prepareFunc != nil { if err := tc.prepareFunc(kubelet); err != nil { t.Fatalf("%s failed preparation: %v", name, err) } } exist := kubelet.podVolumesExist(poduid) if tc.expected != exist { t.Errorf("%s failed: expected %t, got %t", name, tc.expected, exist) } }) } } </ins>

In , we check mount on first level directory in pod volume directories. It is true for many volume plugins, but not for CSI plugin which . cc
/assign
Pardon my ignorance of kubelet source code, but this looks very much similar to an issue we have been running to many times recently, when developing a CSI plugin. The problem is that when CSI volume publish is called and the mount point is not propagated from the CSI plugin pod (either missing bidirectional flag in yaml or not supported by underlaying k8s cluster implementation), the kubelet nevertheless starts a pod using the volume. The volume is not there, so the app writes to root filesystem in fact, leading to all other kinds of problems - i.e. losing data, filling up root fs. IMHO what kubelet should do is to check that the mounted filesystem is indeed visible after publish call and if not it should fail the volume attach request. If this ticket was fixed, would it solve the issue we are seeing or should I open a new ticket? thanks
I think we can potentially add that check but we need to make sure first it can work with ephemeral CSI drivers. Ephemeral drivers would have to "stage" their directories somewhere, like /var/lib/kubelet/plugins/<csi driver/, and then bind mount it to the publish path. cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Hmm... That is a good question. Does csi require a mount to be made or only that the specified directory be created? Will have to go read through the spec...
/remove-lifecycle stale
The only place in the spec I could find was this comment: // For volumes with an access type of mount, the SP SHALL place the // mounted directory at target_path. So I think having Kubelet enforce that check would be ok with the spec, and a good thing to do.
cc CSI spec says that the target path has to be a mount. Shall we go ahead and enforce it then? Are we going to break drivers that may not be doing this?

Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
The issue mentioned here involves two functions . Now with PR , the issue in CheckVolumeExistenceOperation is solved by checking mount path instead of volume path. The only issue here is for getMountedVolumePathListFromDisk which is used during pod deletion. This function was introduced by PR . Now I am questioning whether we need to check the mount point or just need to check pod dirs exist before deleting the pod. Firstly, for some plugins, there might not be mount point. Secondly, without checking mount point, it means that pod cannot be deleted as long as there are pod volume dirs. In normal case, volume dirs should be cleanedup during volume teardown. For cases that kubelet restarts, we have reconciler try to clean up volume mounts if possible. So we can propose a couple of solutions here mount point check. It solves the problem, the downside is that there might be some rare case that reconciler could not clean up dirs after kubelet restarts. check mount point, and also check the next level "/mount" dir. This might still cause problem for some plugins that do not have mounts. cc
Discussed offline with For proposal 1., pod will not be deleted from api server if there are still volume dirs exist. So to make sure it works well, we need to have reliable way to clean up dirs. Normally when kubelet restarts, reconciler will either add it into actual state (if reconstruction works) or try to clean it up directly once unless volume is still in desired state. So if reconstruction does not work, and it fails to clean up during reconciler sync, there is no other chance to clean up the dirs.

Or if volume was in desired state and removed quickly afterwords, the dir might not get the chance to be cleaned up at all. We are thinking to improve this part, we could use mark volume "uncertain" in actual state feature. During reconstruction, we can remove the check for mounted points (since the mount might be gone during node reboots or driver does not have moutns at all). Reconciler could mark those volumes as uncertain (even if it was in desired state) to make sure those volume got cleaned up or remount whenever needed with retries. With this change, I think it is safe to remove the mount point check for pod deletion too.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale


Question: func PodToSelectableFields(pod *api.Pod) labels.Set { return labels.Set{ "name":         pod.Name, <del> "Status.Phase": string(pod.Status.Phase), "Status.Host":  pod.Status.Host, </del> <ins> "status.phase": string(pod.Status.Phase), "status.host":  pod.Status.Host, </ins> } }

We use capitals for Pod fields (ex: Status.Host) and camelCase for Event fields (ex: involvedObject.resourceVersion). We should be consistent. Should it be capitals? camelCase? Or should we be case insensitive?
IMO, we should use camelCase for pod fields, to match the way it's serialized in JSON.
I agree with The goal here is to be consistent with the API, not with the internal Go representation.
I'd like to fix this before v1beta3 is cast in stone.
ok. Event field selectors are correct then. I will update the selectors for pods to make them camelCase.
I am wondering if I should change this only for v1beta3 or for 1 and 2 as well? This will be a breaking change for beta1 and 2 if we decide to do that.
v1beta3 only, though that's hard, we should reconsider. We're contemplating another "breakage day", and I doubt this is widely used.


Question: if err != nil { return false, false, err } <del> for _, existingPod := range allPods { </del> <ins> // Special case: When the topological domain is node, we can limit our // search to pods on that node without searching the entire cluster. if term.TopologyKey == kubeletapis.LabelHostname { pods = nodeInfo.Pods() } for _, existingPod := range pods { </ins> match := priorityutil.PodMatchesTermsNamespaceAndSelector(existingPod, namespaces, selector) if match { matchingPodExists = true

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --It is not a secret that . In my opinion the main reason for the slowness is the flexibility of the inter-pod affinity/anti-affinity rules. However, in most practical scenarios users do not need all the flexibility. For example, people often do not use arbitrary labels in topology key of the affinity rules. They often use node, zone, or region as the topology. Node in particular is used often. That's why I am proposing a fast path for the case that topology key is Node. Today, the affinity/anti-affinity predicate does not have any distinction for topology keys and in the cluster regardless. The logic should recognize "" and scan only pods on the same node for this topology key. Is this a BUG REPORT or FEATURE REQUEST?: FEATURE REQUEST /kind feature /sig scheduling cc/
ref/
/assign
GitHub didn't allow me to assign the following users: misterikkit. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
We found some other simple optimizations that had a significant impact on the affinity predicate. I will upload PRs for those as well.
Reopening this as the performance of anti-affinity still doesn't seem to be satisfactory enough on large clusters (i.e 5k nodes) - and it's causing them to fail to startup. Ref: issue I can help with evaluating any performance improvements if needed.
The change sent improved performance of the predicate, but apparently it is not enough to make our large scale scalability tests happy.
This issue is not about general performance improvment of affinity/anti-affinity. It was to implement a particular improvment which is already implemented. There are other performance improvement issues. I am going to close this one.
Sounds good. Thanks.
How come with kops 1.10.0 I still see all kube-dns pods on the same node because the kube-dns deployment doesn't specify any anti-affinity rule?
I think you are correct. you can double-check with the option to . I have a v1.9 cluster in front of me, and kube-dns has no affinity rules. (I didn't use kops though)


Question: By("Trying to launch a pod that with PodAffinity & PodAntiAffinity setting as embedded JSON string in the annotation value.") labelPodName := "with-newlabels" <del> testPodPath := string(framework.ReadOrDie("test/e2e/testing-manifests/node-selection/pos-with-node-affinity.yaml")) </del> <ins> testPodPath := "test/e2e/testing-manifests/node-selection/pod-with-pod-affinity.yaml" </ins> framework.RunKubectlOrDie("create", "-f", testPodPath, fmt.Sprintf("--namespace=%v", ns)) // check that pod got scheduled. We intentionally DO NOT check that the

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity

setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s-

Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the

annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
can you please take a look at this?
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
- this test is completely broken in serial suite and is P0 for last 3 days.
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
- do you think is the correct owner for this test?

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
- I'm going to make you an owner of this test (), as you're an author if github is to be believed. FYI: assigning random people to tests doesn't work (as can be seen in this example).
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
thanks for reminding, I'm taking look at this.

https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
Found the root cause
- awesome job! Thanks!
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}
https://k8s- Failed: [] SchedulerPredicates [Serial] validates that embedding the JSON NodeAffinity setting as a string in the annotation value work {Kubernetes e2e suite}


Question: $(META_DIR)/$(DEFAULTER_GEN).todo: $(DEFAULTER_FILES) $(DEFAULTER_FILES): $(DEFAULTER_GEN) <del> if [[ "$(DBG_CODEGEN)" == 1 ]]; then          </del> <ins> if [[ "$(DBG_CODEGEN)" == 1 ]]; then       </ins> echo "DBG: defaulter needed $(@D):";   <del> ls -lf --full-time $@ $? || true;         </del> <ins> ls -lf --full-time $@ $? || true;      </ins> fi echo $(PRJ_SRC_PATH)/$(@D) >> $(META_DIR)/$(DEFAULTER_GEN).todo

In the codegenerator we have: So basically we process only packages that are explicitly requested by the user and ignore ones that are imported. So in order to have the conversion func imported by the generator, the package where it lives has to be explicitly imported. So we should basically always pass all input packages to conversion generator. This is problematic, when regenerating conversions, we simply pass to input directories only those that were changed in the meantime: We should fix that - probably by passing all CONVERSIONDIRS as inputs to conversion-gen /kind-bug
I think this caused Not sure if was related yet.
It's the other way around - this is the main bug and root cause and the other that you linked is a consequence.
Also - FYI
I actually already started looking into it - so let me assign this.
is out for review
/triage accepted


Question: which represents all IPs on the host. release: v1.16 file: test/e2e/scheduling/predicates.go <del> - testname: Scheduling, HostPort matching and HostIP and Protocol not-matching codename: '[sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [LinuxOnly] [Conformance]' description: Pods with the same HostPort value MUST be able to be scheduled to the same node if the HostIP or Protocol is different. This test is marked LinuxOnly since hostNetwork is not supported on Windows. release: v1.16 file: test/e2e/scheduling/predicates.go </del> - testname: Pod preemption verification codename: '[sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]'

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: We want to change ownership of a test from scheduler to network Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates that there is no conflict between pods with same hostPort but different hostIP and protocol [Conformance] Since when has it been failing: Testgrid link: https://k8s- Reason for failure: Once merges, we will exclude hostPort test on Windows. This is because of how the hostPort test is structured where we are relying on hostNetwork pod to test the communication between hostPort pods. This will not work on Windows as hostNetwork is not supported. I think it is better to have this test in sig-network and test the communication as scheduler should be responsible only for assigning a pod to node. This also ensures that hostPort on Windows is exercised properly. Anything else we need to know: More context
/sig scheduling /sig network cc
/area conformance /area test /sig windows since windows
/triage accepted


Question: } pod.Spec.Host = machine pod.Status.Host = machine <ins> if pod.Annotations == nil { pod.Annotations = make(map[string]string) } for k, v := range annotations { pod.Annotations[k] = v } </ins> finalPod = pod return pod, nil })

Recap of related tangent in follows. Use case: I need to implement mesos task reconciliation in the kubernetes-mesos scheduler. If it crashes and then restarts, I want to be able to recover things like taskID, slaveID, offerID, etc. so as to rebuild the internal state of the scheduler. Once state is recovered, I can reconcile the against the tasks that the mesos master knows about. If I can't use binding annotations for this, then I need to store state some other (super inconvenient) way. Currently the etcd registry assignPod() implementation drops binding annotations on the floor. It should not, and instead they should be preserved for later access. Proposal: Store binding-related annotations in a new field: Pod.Status.Annotations When setting Pod.Status.Host="{some-host-name}" via assignPod(), binding annotations are copied in from Binding When cloning Pod.Status.Host in UpdatePod() (for an already-scheduled pod), the Pod.Status.Annotations are also cloned. I don't think /bindings has to be a readable apiserver endpoint, as long as the binding metadata is exposed some other way, as proposed above. I'd also like to propose that once binding annotations are "bound" to Pod.Status they become read-only. So they're essentially write-once, just like Pod.Status.Host is meant to be. cc xref
There is one situation where we probably want to clear Host, actually: in the case that the pod can't actually be materialized on the selected node for some reason, such as .
Any objection to a PR implementing my proposal? I'm happy to do the work.
You'll probably want to do it on top of (or after it lands) since that code path is changing significantly.
Though it is also represented in PodStatus, Host is properly a field in the PodSpec. Status must be completely reconstructable based on observations. The proposed Pod.Status.Annotations would not be. Furthermore, I don't think we should introduce another annotations field to Pod. I'd prefer to add a podAnnotations field to Binding, and then merge them into the pod. As for clearing the annotations, I'd prefer to simply fail a pod that was rejected by a node so as not to have to deal with that. Also note the discussion here re. moving binding to a pod subresource:
See also .


Question: }, { api.Pod{ <del> ObjectMeta: api.ObjectMeta{Name: "foo", DeletionTimestamp: &now}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: "foo"}, </ins> Spec:       api.PodSpec{Containers: []api.Container{{Image: "foo:V1"}}}, }, api.Pod{ <del> ObjectMeta: api.ObjectMeta{Name: "foo"}, </del> <ins> ObjectMeta: api.ObjectMeta{Name: "foo", DeletionTimestamp: &now}, </ins> Spec:       api.PodSpec{Containers: []api.Container{{Image: "foo:V1"}}}, }, true,

Honestly I'm not sure if this is a bug or a feature, but didn't expect it. Filing this to track.
I think it's a bug. Nothing in the code base depends on it that I can see (edit: namespace deletion does, but only via an in-server call which I'd like to see fixed), and we don't let other delete-related fields mutate during update (deleteGracePeriodSeconds)


Question: limits: memory: 200Mi requests: <del> cpu: 100m </del> <ins> cpu: 90m </ins> memory: 200Mi volumeMounts: - name: varlog

This causes one component on master to fail in our standard test clusters. Usually it's NPD: cc
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/sig cluster-lifecycle


Question: "k8s.io/kubernetes/test/integration/framework" ) <del> // TestCustomResourceValidatorsWithDisabledFeatureGate test that x-kubernetes-validations work as expected when the // feature gate is disabled. func TestCustomResourceValidatorsWithDisabledFeatureGate(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, genericfeatures.CustomResourceValidationExpressions, false)() server, err := apiservertesting.StartTestServer(t, apiservertesting.NewDefaultTestServerOptions(), nil, framework.SharedEtcd()) if err != nil { t.Fatal(err) } defer server.TearDownFn() config := server.ClientConfig apiExtensionClient, err := clientset.NewForConfig(config) if err != nil { t.Fatal(err) } dynamicClient, err := dynamic.NewForConfig(config) if err != nil { t.Fatal(err) } t.Run("x-kubernetes-validations fields MUST be dropped from CRDs that are created when feature gate is disabled", func(t *testing.T) { schemaWithFeatureGateOff := crdWithSchema(t, "WithFeatureGateOff", structuralSchemaWithValidators) crdWithFeatureGateOff, err := fixtures.CreateNewV1CustomResourceDefinition(schemaWithFeatureGateOff, apiExtensionClient, dynamicClient) if err != nil { t.Fatal(err) } s := crdWithFeatureGateOff.Spec.Versions[0].Schema.OpenAPIV3Schema if len(s.XValidations) != 0 { t.Errorf("Expected CRD to have no x-kubernetes-validatons rules but got: %v", s.XValidations) } }) } </del> // TestCustomResourceValidators tests x-kubernetes-validations compile and validate as expected when the feature gate // is enabled. func TestCustomResourceValidators(t *testing.T) {

This is the tracking issue of promoting CRD validation rules to GA: Listed the blocking issues/features below: [x] [x] [x] [x] Issues be aware of but should not be a GA blocker [ ] [ ] [ ] [ ] [ ] /sig api-machinery /triage accepted
/assign
/cc


Question: ### Setup <del> Download a prebuilt Debian VMDK to be used as base image: </del> <ins> Download a prebuilt Debian 7.7 VMDK that we'll use as a base image: </ins> ```sh <del> wget https://storage.googleapis.com/govmomi/vmdk/kube.vmdk.gz{,.md5} </del> <ins> curl --remote-name-all https://storage.googleapis.com/govmomi/vmdk/2014-11-11/kube.vmdk.gz{,.md5} </ins> md5sum -c kube.vmdk.gz.md5 gzip -d kube.vmdk.gz ``` <del> Upload this VMDK to your vSphere instance: </del> <ins> Import this VMDK into your vSphere datastore: </ins> ```sh export GOVC_URL='user:pass@hostname'

When wgetting to this address during the installation process, this just fails silently. Turns out the issue is that if you take a raw and curl that address you get a TLS error. I've fixed it by changing to , but obviously that's more risky.
Related (filed after an email exchange around wget) I don't have easy access to the vSphere so I can't verify how it relates to my debian install.
Yes, that is exactly the error I was experiencing.
Does curl work? Or is it busted too?
Curl is not installed on the image by default. I'm building a new image based on Debian 7.7 where I pull in curl by default, so we can switch to using that everywhere (instead of an awkward mix of wget and curl).
Thanks for reporting this I hadn't seen this yet. I uploaded a new VMDK that includes curl and updated the deployment scripts to use curl. I verified that deployment works again for me.
I can confirm that this works now with latest build from master on ESX. Probably still won't work with v0.4.3 (even w new image) since the scripts still used wget back then.
We are probably going to fork for v0.5 soon. But I can backport if this trips folks up. sent from tiny keyboard On Nov 12, 2014 9:09 AM, "Ben Smith" wrote:


Question: }, Limits: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("1250m"), <del> v1.ResourceMemory: resource.MustParse("64Mi"), </del> <ins> v1.ResourceMemory: resource.MustParse("128Mi"), </ins> }, }, VolumeMounts: []v1.VolumeMount{

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: node-kubelet-master Which test(s) are flaking: [sig-storage] Downward API volume should provide container's memory limit [NodeConformance] [Conformance] [cos-stable1] [sig-storage] Downward API volume should provide container's cpu limit [NodeConformance] [Conformance] [ubuntu] Testgrid link: https://k8s- Reason for failure: Both of these tests are failing with same reason. Anything else we need to know: links to appreciated links to specific failures in spyglass appreciated <!-- Please see the deflaking doc () for more guidance! --/sig storage
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Hi this issue needs your attention.
Still flaking
Hi can you call attention of the relevant folks here as well ?
For the first test failure posted here on 09/30. The reason is not enough CPU available. The type of error only happened once, it never happen again for this test. For the test failure mentioned, more than half of the tests (150+) failed including this Downward API tests. It happened quite often for ci-kubernetes-e2e-windows-containerd. It seems something wrong with container runtime. The pods are all stuck on pulling image phase, no other events after that and then timed out. Seems containerd for windows have some issues.. So I think this specific test is not considered as flaky test.
Latest failure: This failure includes, lot of other failing test as well. So could, for another major reason. Time Test -- -- 01/12/2020, 06:31:17 [sig-storage] Downward API volume should provide container's memory limit [NodeConformance] [Conformance] Though most flakes are on windows related jobs. But, there are other instances too.

Do you think, above failure is for the same reason.
I am seeing downward api flakes in the Windows tests as well. If I broaden the search a bit it seems to happen for more than just memory limit but also: Test [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's memory request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu limit [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] [sig-storage] Downward API volume should provide container's cpu request [NodeConformance] [Conformance] For Windows they seem to be failing with the following (both docker and containerd): I will can look into this from a Windows perspective. /assign
For Windows, I do not believe this is an issue with the test it's self. I ran this subset of tests continuously over the weekend and got the following results: The is "paging file is too small": I was able to force similar (but not exactly the same) errors by over consuming memory via a test pod. I believe this might be happening due to other tests running on the cluster at the same time. I am looking into this further.
A few tests failed similar to this over the weekend:
FYI , during this saw some of the issues w/ containerd leaks
/sig windows
For the aks-engine Containerd test cases we lowered the number of running to the tests concurrency value we had prior to 1.19 and this has gone away. I am continuing to looking into if this was a regression or if is releated.
We haven't seen this in a long time but it came up today:
Another failure in gce windows test:
another failure here:
/milestone v1.22
another failure:
cc
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close [1]:
Closing this issue. <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen /remove-lifecycle rotten We saw this again in master
Reopened this issue.

<detailsIn response to : /reopen /remove-lifecycle rotten Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone clear
We have a lead this might be related to working set bytes vs commit bytes and the way that windows handles memory.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/sig node
/remove-lifecycle rotten
/remove-sig It looks like sig storage issue. Please reapply if requires sig-node to triage.
This still flakes. An example: are you still working on this?
I haven't looked it from a Linux perspective. From Windows side, it seems this is caused by a mis-match between the commit memory and working set:
Posting some details here, since the issue above has been mentioned. We've been flaking on these downward API tests for quite a while, with errors like these: Logs taken from: Those error logs are actually from the Pod itself, which is supposed to run the agnhost golang binary. It doesn't always happen, but it does frequent enough to be visible in our Windows dashboards (8 failures in the past week, according to: (%20volume)%3F%20should%20provide%20container%27s).

Looking through the downward API E2E tests, we noticed that some memory limits are being set in 8 test cases (). The memory limit set is 64MB, which is a bit low for a Windows Pod, even if it's nanoserver-based (also, the agnhost golang binary is not that small either, it has 44.2 MB on Linux). IMO, we could increase the memory limit to 128MB and we should see fewer flakes.
+1


Question: if err != nil { return err } <del> if !data.State.Running && (kl.minimumGCAge == 0 || time.Now().Sub(data.State.FinishedAt) > kl.minimumGCAge) { </del> <ins> if !data.State.Running && (time.Now().Sub(data.State.FinishedAt) > kl.minimumGCAge) { </ins> dockerData = append(dockerData, data) } }

Things like SyncFrequency should have a lower-bound of acceptable, and that really should be 0ns.
The main point of this is now fixed, though we could be better about input checking all over


Question: // ClientForVersion initializes or reuses a client for the specified version, or returns an // error if that is not possible func (c *clientCache) ClientForVersion(version string) (*client.Client, error) { <ins> if client, ok := c.clients[version]; ok { return client, nil } </ins> config, err := c.ClientConfigForVersion(version) if err != nil { return nil, err } <del> if client, ok := c.clients[config.Version]; ok { return client, nil } </del> client, err := client.New(config) if err != nil { return nil, err

This seems like a bug to me No output, and command exits success... wat. Have to explicitly pass to see expected output. If the client version is greater the server version, it should be making requests based on the server's newest accepted apiversion.
cc


Question: export SECONDARY_RANGE_NAME="pods-default" export STORAGE_BACKEND="etcd3" export STORAGE_MEDIA_TYPE="application/vnd.kubernetes.protobuf" <del> export ETCD_IMAGE=3.5.3-0 export ETCD_VERSION=3.5.3 </del> <ins> export ETCD_IMAGE=3.5.4-0 export ETCD_VERSION=3.5.4 </ins> # Upgrade master with updated kube envs "${KUBE_ROOT}/cluster/gce/upgrade.sh" -M -l

Please publish 3.5.4-0 image and update to to benefit from fix for , while still keeping v3.5.3 critical correctness fixes (, ). For recent PRs for update to etcd v3.5.3 see and
/sig release /sig api-machinery /kind bug
/cc /triage accepted
Hey, Can I work on this ? I'm a beginner, but all the good first issues are mostly assigned. Can you guide me on doing this ?
I am working on this. /assign
Hi I would like to work on Make sense to backport in v1.22, v1.23 & v1.24? Thanks!!
hi, it makes sense to me, but requires approval from dependency maintainers. i can only approve the kubeadm changes.
was this backported ?


Question: Once you have that service description, you can create the service with the `cloudcfg` cli: <del> ```shell cluster/cloudcfg.sh -c redis-master-service.json create /services </del> <ins> ```js $ cluster/cloudcfg.sh -c examples/guestbook/redis-master-service.json create /services Using master: kubernetes-master (external IP: 1.2.3.4) { "id": "redismaster", "port": 10000, "labels": { "name": "redis-master" } } </ins> ``` Once created, the service proxy on each minion is configured to set up a proxy on the specified port (in this case port 10000). ### Step Three: Turn up the replicated slave pods. <del> Although the redis master is a single pod, the redis read slaves are a 'replicated' pod, in Kubernetes, a replication controller is responsible for managing multiple instances of a replicated pod.  Create a file named `redis-slave-controller.json` that contains: </del> <ins> Although the redis master is a single pod, the redis read slaves are a 'replicated' pod. In Kubernetes, a replication controller is responsible for managing multiple instances of a replicated pod. </ins> <del> ```javascript </del> <ins> Create a file named `redis-slave-controller.json` that contains: ```js </ins> { "id": "redisSlaveController", "desiredState": {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I am running an admission webhook using If there it has been hours since the last mutating request to the webhook, then the next mutating request will take many seconds to be processed. This may even take long enough to have the request time out. What you expected to happen: A request takes approximately the same amount of time to be processed regardless of how long it has been since the last request. How to reproduce it (as minimally and precisely as possible): Create an API server for handling a new API group. Wait a couple hours. Send a CREATE request to the API server. Observe that the CREATE request takes a few seconds to complete. Anything else we need to know?: This is the offending code. It must loop through once for every second since the last request. Environment: Kubernetes version (use ): 1.19.0
/sig api-machinery
I thought we'd already merged a fix for this from before the release?
I assume that the fix was supposed to be From my understanding, those changes only are in effect if the apiserver is running with priority and fairness enabled. In my situation, I do not have that enabled.
Ack, if we have the problem in both places we need the fix in both places :)
/triage accepted
I think we have fixed this in , do you guys agree?


Question: "labels.go", "legacy.go", "security_context.go", <ins> "security_context_others.go", "security_context_windows.go", </ins> ], importpath = "k8s.io/kubernetes/pkg/kubelet/kuberuntime", deps = [

Hello, What happened: When we apply PodSecurityPolicy with rule: WIndows containers are still blocked instead we put: the pod is still blocked with logs: What you expected to happen: the Container should start How to reproduce it (as minimally and precisely as possible): Create Psp disableling Root Users: Create a pod Windows with runAsUserName: Anything else we need to know?: Environment: Kubernetes version (use ): v1.18.3 Cloud provider or hardware configuration: OS (e.g: ): CentOS Linux 7 (Core) / windows server 2019 version 1809 Kernel (e.g. ): 3.10.0-514.10.2.el7.x86_64 Others: /sig windows
/sig windows
/assign I will test it and then post the test results
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale


Question: // RegisteredWatchers is a number of currently registered watchers splitted by resource. RegisteredWatchers = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ <del> Name:           "apiserver_registered_watchers", Help:           "Number of currently registered watchers for a given resources", StabilityLevel: compbasemetrics.ALPHA, </del> <ins> Name:              "apiserver_registered_watchers", Help:              "Number of currently registered watchers for a given resources", StabilityLevel:    compbasemetrics.ALPHA, DeprecatedVersion: "1.23.0", </ins> }, []string{"group", "version", "kind"}, )

apiserverlongrunninggauge and apiserverregisterwatchers are kinda redundant . We discussed this in our SIG instrumentation meeting and while it's clear that apiserverlongrunninggauge, the name does not conform to instrumentation guidelines (it was created before those guidelines existed). What we want to do is deprecate both metrics and create a new one which actually does conform to our naming conventions probably something like . has volunteered to undertake this. /cc /sig instrumentation
/assign
GitHub didn't allow me to assign the following users: yan-lgtm. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/triage accepted
Hello! Have you still working on this issue? I am new to the community, and can I try it?
Hey I the deprecate in If you want to add the new metric, feel free to do so.
Can I add the new metric? If I can, can you give me some advice about the new metric? Thank you~
/assign


Question: <ins> # Maintainers People responsible for ports of Kubernetes to different environments. CC at least one maintainer on relevant issues and PRs. ## OS Distributions * RedHat, Fedora: [Clayton Coleman](https://github.com/smarterclayton), [Derek Carr](https://github.com/derekwaynecarr), [Scott Collier](https://github.com/scollier) * CoreOS: [Kelsey Hightower](https://github.com/kelseyhightower) ## Providers / Environments * GCE: [Brendan Burns](https://github.com/brendandburns), [Joe Beda](https://github.com/jbeda), [Daniel Smith](https://github.com/lavalamp), [Tim Hockin](https://github.com/thockin) * Azure: [Jeff Mendoza](https://github.com/jeffmendoza) * Vsphere: [Pieter Noordhuis](https://github.com/pietern) * Rackspace: [Ryan Richard](https://github.com/doublerr) * Ovirt: [Federico Simoncelli](https://github.com/simon3z) * Local: [Derek Carr](https://github.com/derekwaynecarr) * Vagrant: [Derek Carr](https://github.com/derekwaynecarr) </ins>  No newline at end of file

We need to be very clear to users who are trying k8s out where they can turn for help. It's unfortunate when a user is having trouble with cloud provider XYZ and they ask for help on IRC and nobody answers.


Question: WORKDIR /workspace RUN apt-get -o Acquire::Check-Valid-Until=false update && apt-get install -y rsync <ins> # file is used when uploading test artifacts to GCS. RUN apt-get install -y file </ins> # libapparmor1 is needed for docker-in-docker. <del> RUN apt-get install -y file libapparmor1 </del> <ins> RUN apt-get install -y libapparmor1 # netcat is used by integration test scripts. RUN apt-get install -y netcat-openbsd </ins> RUN mkdir -p /go/src/k8s.io/kubernetes RUN ln -s /go/src/k8s.io/kubernetes /workspace/kubernetes

It seems like an issue similar to the problem with API server. I'll extend the timeout in the similar fashion. cc
Extending timeout didn't help. It's probably something else.
cc
I think the logs make it pretty clear what went wrong: or does one of you have time to fix this?
The Jenkins docker test image probably needs netcat to be installed. Should be an easy fix; I can take this.
Thanks!
Ugh... neverending story.
This seems like a different problem.
Oh, I see, this actually is the same, but harder: I think the most obvious thing to do is to let kubectl select the port and report back?
reports more occurances. Netcat somehow is no longer installed?
those two are older builds, using , which didn't have netcat yet.
ah thanks!
are you still working on this? Would you like me to take it?
I'm not right now, sorry, feel free to take it.


Question: if verb == "LIST" && strings.ToLower(request.QueryParameter("watch")) == "true" { verb = "WATCH" } <del> Monitor(&verb, &resource, utilnet.GetHTTPClient(request.Request), rw.Header().Get("Content-Type"), delegate.status, now) </del> <ins> Monitor(&verb, &resource, cleanUserAgent(utilnet.GetHTTPClient(request.Request)), rw.Header().Get("Content-Type"), delegate.status, now) </ins> }) } <ins> func cleanUserAgent(ua string) string { if strings.HasPrefix(ua, "Mozilla/") { return "Browser" } return ua } </ins> type responseWriterDelegator struct { http.ResponseWriter

Some of the exported prometheus metrics, e.g. include high-cardinality fields like the User-Agent. This creats a large number of time-series on the prometheus server is ususally not advised to do. Please consider reducing the cardinality of the exported metrics.


Question: // ProbePodToPodConnectivity runs a series of probes in kube, and records the results in `testCase.Reachability` func ProbePodToPodConnectivity(k8s *kubeManager, model *Model, testCase *TestCase) { <del> numberOfWorkers := 3 // See https://github.com/kubernetes/kubernetes/pull/97690 </del> allPods := model.AllPods() size := len(allPods) * len(allPods) jobs := make(chan *ProbeJob, size) results := make(chan *ProbeJobResults, size) <del> for i := 0; i < numberOfWorkers; i++ { go probeWorker(k8s, jobs, results) </del> <ins> for i := 0; i < model.GetWorkers(); i++ { go probeWorker(k8s, jobs, results, model.GetProbeTimeoutSeconds()) </ins> } for _, podFrom := range allPods { for _, podTo := range allPods {

<!-- Please only use this template for submitting enhancement requests --What would you like to be : Ive done preliminary testing with on calico powered clusters on some windows variants and found that the probing seems to work. Can we add wrapper to netpol suite so that we can run it on windows clusters ? Why is this needed: We need this because otherwise, CNIs such as EKS will give us This error is caused when, for example, pods that are landing on a windows node havent defined their target platform. Somehow i guess pods which are landing on linux nodes dont have this issue. Its also needed for the obvious reason that we probably want separate linux and windows test suites for policies, in general, even though we expect the logic to be identical (modulo hostports , which we dont currently test)
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/assign
grepping around, I can't figure out where those are supposed to go. Can you give me pointers?
(resolved in slack, matts digging into this)
ok, these are running now on eks, lots of failures. will dig
/assign
the netpol are not being applied for the ns: Instead of failing because should not be able to access , the connection succeeds.


Question: } if !s.LeaderElection.LeaderElect { <del> run(nil) </del> <ins> stopCh := make(chan struct{}) defer close(stopCh) run(stopCh) </ins> panic("unreachable") }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: If a 1.8.x kubernetes master with a single controller-manager is configured with --leader-elect=false, deleting a deployment through the API with v1meta.DeletePropagationForeground doesn't work. The deployment has its deletionTimestamp set and a foregroundDeletion finalizer is , but the pod, rc and deployment are not deleted. What you expected to happen: The pod, rc and deployment should be deleted when the controller-manager uses --leader-elect=false. How to reproduce it (as minimally and precisely as possible): I used kubeadm with calico networking, and modified the controller-manager configuration via sudo sed -i /etc/kubernetes/manifests/kube-controller- -e '/leader-elect/ s/true/false/' and then ran to create and delete deployments. Anything else we need to know?: I suspect the problem is related to this change: It appears to require leader election before the GC controller will finish initialization. Looking at controller-manager logs with leader-elect=false, I saw this log entry from "GraphBuilder running" I did not see the "started %d new monitors, %d currently running" log entry at the end of the startMonitors() function. That seems consistent with the GC controller waiting for "<-gb.informersStarted" Environment: Kubernetes version (use ): tested with 1.8.2 and 1.8.4 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): vagrant virtualbox with the ubuntu/xenial64 box Kernel (e.g. ): Install tools: kubeadm Others:
cc Have you seen this in Openshift?
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this problem occurs because initialiuzes the stop channel top nil when leader-elect=false. This causes garbage collector graph_builder to exit early. The following release-1.8 patch seems to fix the problem.

creates a dummy channel which will never be closed and allows the garbage collector graph builder to work like it does when leader-elect is true. [Edit - just noticed that my initial patch had some unexpected differences]
do you really need to remove the following code?
That change was not intended. I also noticed that after I saved my change. I redid the patch and edited my previous comment to reflect that.
Agree with the diagnosis and fix, are you going to open a PR?
Yes, I can do that. Reading your PR guidelines and ours.
Will I need to go through to get this into the release-1.8 and release-1.9 branches?


Question: return *typed.Spec.Replicas, nil } return 0, nil <ins> case *extensions.DaemonSet: return 0, nil </ins> case *batch.Job: // TODO: currently we use pause pods so that's OK. When we'll want to switch to Pods // that actually finish we need a better way to do this.

[sig-apps] Daemon set [Serial] should retry creating failed daemon pods [Conformance] [sig-apps] Daemon set [Serial] should rollback without unnecessary restarts [Conformance] [sig-apps] Daemon set [Serial] should run and stop complex daemon [Conformance] [sig-apps] Daemon set [Serial] should run and stop simple daemon [Conformance] [sig-apps] Daemon set [Serial] should update pod when spec was updated and update strategy is RollingUpdate [Conformance] Please see: https://k8s-,%20master%20(dev) https://k8s-,%20master%20(dev)
can you please take a look? I see at least one change in this range that probably caused this?
/sig apps /priority failing-test
/milestone v1.11
/kind bug /priority critical-urgent
cc
thanks for tracking and opening this issue.
cc and
These tests are also failing in sig-release-master-blocking kubeadm-gce jobs https://k8s- This is release blocking as we are entering Code slush on Tuesday 5/29.
cc
Fix in
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details


Question: HostPort:      &pm.HostPort, ContainerPort: &pm.ContainerPort, Protocol:      &proto, <ins> HostIP:        pm.HostIp, </ins> }) } if config.GetLinux().GetSecurityContext().GetNamespaceOptions().GetNetwork() == runtimeapi.NamespaceMode_NODE {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm using a proxy-pod to forward both and privileged ports to an service using the following configuration: However, when I start this service, it somehow highjacks the ports for other host IPs as well. I have a legacy nginx-proxy running on the same server, inside docker bound on specific non-conflicting IPs, but as soon as I start the pod above, it breaks SSL termination of the existing nginx proxy (outputing the "Fake ingress certificate"). Existing docker-compose: Name Command State Ports
nginxproxy nginx -g daemon off; Up 5.135.123.160:443-443/tcp,5.135.123.162:443-443/tcp, 5.135.123.160:80-80/tcp,5.135.123.162:80-80/tcp nginxproxydockergen /bin/sh -c /usr/local/bin/ ... Up nginxproxyletsencrypt /bin/bash /app/entrypoint. ... Up [...] Server certificate: subject: start date: Jun 11 21:55:37 2018 GMT expire date: Sep 9 21:55:37 2018 GMT subjectAltName: host "" matched cert's "" issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3 SSL certificate verify ok.

iptables -t nat -nvL
I've flushed/dropped all iptables then reboot and again to make sure I had a clean state. Here is the iptables snapshots along the setup: Can confirm that SSL breaks for other IPs as soon as I start the pod proxy (last step). Looks like the culprit is . Destination should be and not . : Does work if I manually patch the IPtables rules: Which gives: Is there a way to automate this kind of patching to make sure the rules are not reverted by kubernetes? Thanks for the kind help :)
Thx for the reply about the iptable rules. According to the describe from u, I guess the cni plugin u used dont use the pod's data clearly. Maybe in the cni flannel,The flannel just change the rule by the hostport. And it think all the ip packets in the host must do the dnat and snat. I think flannel dont catch the hostip in the pod's yaml.
Hi mgcrea. Maybe i know why this happen. Maybe It is not the flannel's problem. I will try to fix this. /assign liucimin
GitHub didn't allow me to assign the following users: liucimin. Note that only and repo collaborators can be assigned. <detailsIn response to : Hi mgcrea. Maybe i know why this happen. Maybe It is not the flannel's problem. I will try to fix this. /assign liucimin Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
any update on this?

Did you ever find out the root cause? /remove-lifecycle rotten
Yes,I have push a pr in k8s.
v1.11.5 also the problem. Has Version 1.12 been fixed? thanks.
The pr is not merge yet. So the problem is also in v1.12 and v1.13


Question: fi echo "Uploading to Amazon S3" <del> if ! aws s3 ls "s3://${AWS_S3_BUCKET}" > /dev/null 2>&1 ; then </del> <ins> if ! aws s3api get-bucket-location --bucket ${AWS_S3_BUCKET} > /dev/null 2>&1 ; then </ins> echo "Creating ${AWS_S3_BUCKET}" # Buckets must be globally uniquely named, so always create in a known region

When I run: I get the following error: Waiting a bit and running got past this point because the bucket was ready the second time around, but the getting-started process should be more robust.
I experienced the same issue
We do try to query for the bucket existing, but perhaps S3 is even more eventually consistent than I thought! I did observe that we were very trusting about the user providing the correct S3 region, and have tried to fix this in . I don't think that was the problem here though (if you make_bucket, you can't have a region mismatch...) I guess we could retry the sync a few times; might help with e.g. a network interruption during the sync also.
I believe this to have been fixed by . I'm going to close, but if it happens again please do reopen.
FYI I got this with the latest k8s as of 10/13/15: I believe this is k8s 1.0.6 but I just tore down my AWS cluster and apparently kubectl needs to talk to the now-terminated master instance. I ran to get k8s. Waiting and running again doesn't show the issue since the bucket is fully created and accessible. I just provisioned a Vagrant cluster in hopes that would help show what version I'm using. Here's the output of :


Question: } func verifyStringArrayEqualsAnyOrder(t *testing.T, actual, expected []string) { <del> invalid := len(actual) != len(expected) if !invalid { for _, exp := range expected { found := false for _, act := range actual { if exp == act { found = true break } } if !found { t.Errorf("Expected element %q not found in %#v", exp, actual) } } } if invalid { t.Errorf("Expected: %#v, Actual: %#v", expected, actual) </del> <ins> var act, exp []string copy(act, actual) copy(exp, expected) sort.StringSlice(act).Sort() sort.StringSlice(exp).Sort() if !reflect.DeepEqual(exp, act) { t.Errorf("Expected(sorted): %#v, Actual(sorted): %#v", exp, act) </ins> } }

Seeing random failures.
Looks like lately refactory makes docker API requests are not deterministic. Will look into it.
Any update? It seems to be failing a lot.
I just looked into this and found that is the only test that there are two pods in the which is fed to I think podworker was syncing the two pods in parallel, this causes non-deterministic behaviour I have tried , no errors. , error happens continuously. I can try to fix this :)
Yes, that is what I believe here, but never have time working on it. Thanks for looking into this, and re-assign it to you.
I saw your old PR for this, I think we still need to keep the parallel sync here. Just need to create a for this case :)


Question: // TestListContainers creates several containers and then list them to check // whether the correct metadatas, states, and labels are returned. func TestListContainers(t *testing.T) { <del> ds, _, _ := newTestDockerService() </del> <ins> ds, _, fakeClock := newTestDockerService() </ins> podName, namespace := "foo", "bar" containerName, image := "sidecar", "logger"

https://k8s- Pod has a restart count 1 FYI, all relevant messages about this pod in the From the sync loop timeline, some containers died. Check the log again: The sandbox was GC'd right after it was created. This led to kubelet killing all the containers in the pod since the sandbox needed to be recreated. Need to check whether this was caused by the container state emulated by fakedockerclient, or by a real bug. /cc
/cc FYI.
kuberuntime GC routine has a minAge protection such that it can't GC any pod sandbox that was created no more than 30 seconds ago). This doesn't seem to have be respected from the log. I suspect that the fakedockerclient was not returning a proper timestamp.
GC logic might be faulty. I see a similar failure in https://k8s- EDIT: The node name is The sandbox was created and removed within 2 seconds
- by looking at the logs, can you tell if there's an easy repro?
So given that this happens also in non-kubemark suites, it seems like a release blocker to me. Anyway - - do you think it's releated to number of pod on a node, or it's just that it happens from time to time, and in scalability suites (no matter if it's kubemark or not) we have more of those tries (due to scale)?
I think the timestamp comparison logic might be wrong in the sandbox garbage collector. This would only happen (in my theory) when the the snapshot of all containers/sandboxes on the node are taken when: sandbox is created but not running (started) yet. other containers in the pod haven't not been created. In this case, the fail-safe (minAge) kicks in, and apparently failed in the tests discussed above. This would more likely to happen when during batch creation of more pods. kubelet would usually heal itself by restarting all container in the pod (which has the undesirable side-effect of killing the user containers of course). I think other tests might not catch this because 1) they don't perform large batch creation, and 2) they don't check the restart count.
Oh the above comment was completely based on my theory.

Reproducing this deterministically on a working node may need some hacking in the kubelet code (e.g., inject a long enough sleep after sandbox creation). I will try writing a unit test first to verify the behavior.
Can't reproduce this in my cluster with artificially injected latency... Adding more debug information in For people who are interested in checking the logs and identify the victim pod, here are the instructions: Download the artifacts: E.g., Find the (density) pod with restart count 0:
Didn't reproduce the problem either. I think so. GC can't distinguish just-created sandboxes from failed sandboxes. Especially when the time from sandbox creation to ready is longer than GC's minAge. I'm wondering should runtime (e.g. dockershim) filter out those created-but-not-ready sandboxes from sandbox list? or introduce another sandbox state in CRI?
has been merged 11 hours ago and scalability suite hasn't flaked since. Will monitor and see if there is any need to reopen.
OK. But because , there are still potential problems under big pressure.
First, this issue is no longer a release blocker after the fix. Let's open a separate issue to discuss potential improvement, instead of commenting on a closed issue :) Also, FWIW, we used the same logic in the old docker manager, so this is also not a regression.
Agreed. Opened for this.