Question: typ reflect.Type } <ins> // unexportedTypePanic is thrown when you use this DeepEqual on something that has an // unexported type. It indicates a programmer error, so should not occur at runtime, // which is why it's not public and thus impossible to catch. type unexportedTypePanic []reflect.Type func (u unexportedTypePanic) Error() string { return u.String() } func (u unexportedTypePanic) String() string { strs := make([]string, len(u)) for i, t := range u { strs[i] = fmt.Sprintf("%v", t) } return "an unexported field was encountered, nested like this: " + strings.Join(strs, " -> ") } func makeUsefulPanic(v reflect.Value) { if x := recover(); x != nil { if u, ok := x.(unexportedTypePanic); ok { u = append(unexportedTypePanic{v.Type()}, u...) x = u } panic(x) } } </ins> // Tests for deep equality using reflected types. The map argument tracks // comparisons that have already been seen, which allows short circuiting on // recursive types. func (e Equalities) deepValueEqual(v1, v2 reflect.Value, visited map[visit]bool, depth int) bool { <ins> defer makeUsefulPanic(v1) </ins> if !v1.IsValid() || !v2.IsValid() { return v1.IsValid() == v2.IsValid() }

This tree has a commit that demonstrates two obviously different objects that should not pass the semantic DeepEqual but do.
What shall we do with this issue for triage? Labels? Priority?
P2 for when he returns , Satnam Singh wrote:
OK I am now sufficiently worried about this. Fixes incoming.


Question: "authentication_test.go", "authn_audit_test.go", "authorization_test.go", <ins> "cachecontrol_test.go", </ins> "impersonation_test.go", "requestinfo_test.go", ],

The kube-apiserver is protected, so it's responses should not be cached. The only known exception is for openapi data which is not sensitive, which is large, and which has etags. /assign
/sig api-machinery


Question: t.Fatalf("metadata.resourceVersion not found") } <del> // this call waits for the resourceVersion to be reached in the cache before returning. // We need to do this because the patch gets its initial object from the storage, and the cache serves that. // If it is out of date, then our initial patch is applied to an old resource version, which conflicts // and then the updated object shows a conflicting diff, which permanently fails the patch. // This gives expected stability in the patch without retrying on an known number of conflicts below in the test. // See https://issue.k8s.io/42644 _, err = noxuResourceClient.Get(context.TODO(), "foo", metav1.GetOptions{ResourceVersion: patchedNoxuInstance.GetResourceVersion()}) if err != nil { t.Fatalf("unexpected error: %v", err) } </del> // Scale.Spec.Replicas = 7 but Scale.Status.Replicas should remain 0 gottenScale, err := scaleClient.Scales("not-the-default").Get(context.TODO(), groupResource, "foo", metav1.GetOptions{}) if err != nil {

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-integration Which test(s) are flaking: TestSubresourcePatch Testgrid link: Reason for failure: Unknown. A diff with the original object would be helpful. Anything else we need to know: /sig api-machinery
OK Trying to reproduce, see what's going on.
Clearly this is because SSA is updating the tracked entry's time. It happens when the two request don't happen in the same second. Having a between the two requests makes this into a definitive error.
/wg apply
This seems to be because of . A repeated patch like will always trigger a new managedFields entry to be created, since it's always going to change the object (status or spec will be wiped depending on the endpoint), so the patch will always cause the object to change, which will always cause the entry to be created.
/assign
/assign
/assign
Still seeing this, bumped into it over in
deflake in while the underlying issue is being worked on in


Question: { input: watch.Event{Object: nil}, pods:  []api.BoundPod{}, <del> fail:  true, </del> <ins> fail:  false, </ins> }, { input: watch.Event{Object: &api.BoundPods{}},

To reproduce: Start a cluster via hack/local-up- Start some pods Kill the cluster "docker ps -a" should show the pod containers running still Start the cluster again "etcdctl ls /registry" should show that there is no boundpods directory -- only minions and services "docker ps -a" should show the old pod containers still running Start a new pod After, say, 10s, "docker ps -a" should show the old containers dead /cc
I suspect the cause of this might be : which guarantees that the pod sync logic is never called if is nil. Need to get a repro to confirm.


Question: "//pkg/kubelet/apis/config:go_default_library", "//pkg/kubelet/cm/cpuset:go_default_library", "//pkg/kubelet/types:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", </ins> "//staging/src/k8s.io/apimachinery/pkg/util/errors:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/validation:go_default_library", "//staging/src/k8s.io/apiserver/pkg/util/feature:go_default_library",

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --See for more context. What happened: When you set and , you'd expect container cfs quota period to be . However, unless you also set in the feature flags, the quota value will be silently ignored. What you expected to happen: At least a warning be logged if the feature is not enabled but is set. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /assign karan
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig node Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ) var _ = SIGDescribe("[Feature:Windows] Kubelet-Stats [Serial]", func() { <del> f := framework.NewDefaultFramework("kubelet-stats-test-windows") </del> <ins> f := framework.NewDefaultFramework("kubelet-stats-test-windows-serial") </ins> ginkgo.Describe("Kubelet stats collection for Windows nodes", func() { ginkgo.Context("when running 10 pods", func() {

Lets put a metrics CI job in place that isnt . Investigate our test coverage of metrics-server See why our CI jobs dont test metrics-server Confirm that we are doing enough in is a good test, lets make it part of all our containerd jobs by removing the Serial tags
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig windows
/good-first-issue
This request has been marked as suitable for new contributors. Please ensure the request meets the requirements listed . If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
while were at it make that iterations a non-magic number
while were at it make sure the metrics stats are non-zero , so that we catch regressions turns out we DO run SERIAL jobs in the sig-windows dashboard either we dont collect enough metrics, so the tests pass, or we do, and we dont watch the dashboards :) https://k8s- <-- check the sig-windows dashboard...
this test starts 10 pods on a node at once and run into issues starting 10 pods on a single node when this was run in parallel with other jobs. Maybe we should consider adding a different metrics job that just starts 1-3 pods and does deeper validation of the metrics retruned?
Hey, is this issue still open? I would like to work on this, I am new to open source
We cannot de-serialize the existing metrics job due to its nature. We can however create new test cases that do not need to be run in parallel and do deeper validation of metrics like Jay suggested above. I'm not sure if is still working on the issue. If not feel free to work on this. Thanks!

/unassign
/retitle Add non-serialized metrics tests for Windows
do we have to make an another var _ with [Serial] not included in it and run 3 pods instead of 10 in it. Help needed, am I thinking correct here.


Question: } client := clientsetfake.NewSimpleClientset() <del> if tt.errOnCreate != nil { client.PrependReactor("create", "configmaps", func(action core.Action) (bool, runtime.Object, error) { return true, nil, tt.errOnCreate }) } </del> // For idempotent test, we check the result of the second call. <del> if err := UploadConfiguration(cfg, client); !tt.updateExisting && (err != nil) != tt.errExpected { t2.Fatalf("UploadConfiguration() error = %v, wantErr %v", err, tt.errExpected) </del> <ins> if err := UploadConfiguration(cfg, client); err != nil { t2.Fatalf("UploadConfiguration() error = %v", err) </ins> } if tt.updateExisting { <del> if tt.errOnUpdate != nil { client.PrependReactor("update", "configmaps", func(action core.Action) (bool, runtime.Object, error) { return true, nil, tt.errOnUpdate }) } if err := UploadConfiguration(cfg, client); (err != nil) != tt.errExpected { </del> <ins> if err := UploadConfiguration(cfg, client); err != nil { </ins> t2.Fatalf("UploadConfiguration() error = %v", err) } }

What happened: Running fails consistently with timeout errors What you expected to happen: Unit tests should complete without any errors. If this was a configuration problem (e.g. unsupported go version), then I should see a meaningful error rather than just a timeout error. How to reproduce it (as minimally and precisely as possible): Run Anything else we need to know?: This issue has been present in my system for months. Environment: Kubernetes version (use ): master Cloud provider or hardware configuration: OS (e.g: ): Ubuntu 20.04 LTS (Focal Fossa) Kernel (e.g. ): Linux desktop 5.4.0-37-generic -Ubuntu SMP Wed Jun 3 18:57:02 UTC 2020 x8664 x8664 x86_64 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others: go version go1.13.12 linux/amd64 (snap)
/sig testing
/assign
Issues targeted at the specific problematic packages and responsible SIGs will be more effective in getting this addressed
/remove-sig testing Unless we expect some broad testing change (like reconfiguring go test) this isn't SIG testing. This is SIG storage, node, and cluster lifecycle
/sig node /sig cluster-lifecycle /sig storage tagged the right sigs, though per-package issues are far more likely to drive getting this fixed
fix for cmd/kubeadm/app/phases/uploadconfig:
I'm going to open two more issues for the other two packages and work on them
sounds good, in that case please close this one.
New issues for reconciler and csi packages: ,
Closing. This problem has come up in the past, the solution is to just increase the timeout. I missed this the first time I read the doc.


Question: } if o.Local || o.DryRun { <del> return o.PrintObject(o.Mapper, info.Object, o.Out) </del> <ins> if err := o.PrintObject(o.Mapper, info.Object, o.Out); err != nil { return err } continue </ins> } obj, err := resource.NewHelper(info.Client, info.Mapping).Patch(info.Namespace, info.Name, types.StrategicMergePatchType, patch.Patch)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: my yaml: exec kubectl set resources command,just update first object: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration**: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
/sig cli
/kind bug
/assign


Question: "readOnly":         "Mounted read-only if true, read-write otherwise (false or unspecified). Defaults to false.", "mountPath":        "Path within the container at which the volume should be mounted.  Must not contain ':'.", "subPath":          "Path within the volume from which the container's volume should be mounted. Defaults to "" (volume's root).", <del> "mountPropagation": "mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationHostToContainer is used. This field is alpha in 1.8 and can be reworked or removed in a future release.", </del> <ins> "mountPropagation": "mountPropagation determines how mounts are propagated from the host to container and the other way around. When not set, MountPropagationHostToContainer is used. This field is beta in 1.10.", </ins> } func (VolumeMount) SwaggerDoc() map[string]string {


Question: defer cc.EventBroadcaster.Shutdown() // Setup healthz checks. <del> var checks []healthz.HealthChecker </del> <ins> var checks, readyzChecks []healthz.HealthChecker </ins> if cc.ComponentConfig.LeaderElection.LeaderElect { checks = append(checks, cc.LeaderElection.WatchDog) <ins> readyzChecks = append(readyzChecks, cc.LeaderElection.WatchDog) </ins> } <ins> readyzChecks = append(readyzChecks, healthz.NewShutdownHealthz(ctx.Done())) </ins> waitingForLeader := make(chan struct{}) isLeader := func() bool {

Quoted from 's Splitting healthz/ into livez/ and readyz/
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/sig scheduling
I read all the content of the PR comment and am willing to work on this issue Do we need to continue some discussion for this issue
Hi , thanks for offering your help. But I was working in progress before knowing you can work on this. However, please feel free to contribute the similar issue
/assign
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale


Question: return fmt.Errorf("failed to listen on %q: %v", endpoint, err) } <del> return f.server.Serve(l) </del> <ins> go f.server.Serve(l) return nil </ins> } // Stop stops the fake remote runtime.

A race condition between fakeremote.Start() is spawned as a goroutine to listen for connections, and its caller assumes that the listener will immediately be ready to accept connections. This leads to intermittent test failures if the caller attempts before the fake remote runtime is ready to listen. The code needs to be refactored so that the fake remote returns from listen, and then spawns the server as a goroutine and returns. This can be reproduced by running the remoteruntimetest in a loop; it fails periodically. Failure can be accelerated by putting a time.Sleep() in fake.Start() immediately before the call to util.CreateListener; the failure is then effectively deterministic. Fix tested by running about 150,000 iterations without observing any failures.
/sig pod
Reference initial issue:
/sig node


Question: sed -i -e "s@{{ *etcd_protocol *}}@$etcd_protocol@g" "${temp_file}" sed -i -e "s@{{ *etcd_apiserver_protocol *}}@$etcd_apiserver_protocol@g" "${temp_file}" sed -i -e "s@{{ *etcd_creds *}}@$etcd_creds@g" "${temp_file}" <ins> sed -i -e "s@{{ *etcdctl_certs *}}@$etcdctl_certs@g" "${temp_file}" </ins> sed -i -e "s@{{ *etcd_apiserver_creds *}}@$etcd_apiserver_creds@g" "${temp_file}" sed -i -e "s@{{ *etcd_extra_args *}}@$etcd_extra_args@g" "${temp_file}" <del> sed -i -e "s@{{ *etcd_livenessprobe_port *}}@$etcd_livenessprobe_port@g" "${temp_file}" </del> if [[ -n "${ETCD_VERSION:-}" ]]; then sed -i -e "s@{{ *pillar.get('etcd_version', '(.*)') *}}@${ETCD_VERSION}@g" "${temp_file}" else

Currently, etcd livenessProbe is defined here: This means that it proves /health endpoint 3 times (default failureThreshold value) and if all of them fails, it restarts etcd server. /health endpoint returns "unhealthy" if one of the conditions is met (): there is an active Alarm (there are two kinds of alarm: NOSPACE and CORRUPT) there is no raft leader the latency of a QGET request doesn't exceed 1s The problem is that in most of those cases, etcd's restart isn't a right behavior: if there is NOSPACE alarm, the restart will not free that space if there is no raft leader, it means that it cannot talk to other etcd members, so restart will not fix them if the request latency 1s, the etcd cluster is overloaded which is bad, but after restart we will get similar amount of requests (if not more), so restart isn't a right action e.g. in we see that event-etcd gets restarted by kubelet when it's overloaded. If we do not restart etcd at that point, we will likely simply finish work to be done and heal ourselves. I think we should change livenessProbe to detect cases where restart will actually help. /cc
I think that's not necessary true, that may potentially help sometimes. But I agree that's probably pretty rare. /sig api-machinery
Do we have an analysis of why the 3 healthchecks in row failed in Was it timeout, timeout, timeout or something else? Do you say that if etcd is overloaded then restarting it will help? Wouldn't it be even more overloaded and run into restart-loop because it will never become /healthy in time again?
Sorry, typo. I tried to say that in none of those cases restart will help (including 1s latency), but in fact I wrote something else. I changed initial post.
should fix etcd such that it does not declares as unhealthy as a result of overload (while its still making a progress). Ideally the not-healthchecking traffic should be throttled / load-shed, such that healthcheck stays responsive. this is implemented, I would consider giving higher timeout (e.g. 30sec), such that HC request have higher chance to reach beginning of the processing queue and get responded.

is a usual disjoint between 2 purpuses of healthchecking: a) autohealing -by restart -where we need to be very conservative about timeouts, as cost is high. b) measuring SLO & alerting -where we want to measure customer's happiness, and we cannot consider request waiting for 15s+ seconds as healthy/happy state. The lower tolerance more represents customers needs. In practical application we should optimize etcd healthcheck for a), while b) should be based on prometheous metrics.
This is basically the difference between "livez" and "readyz". "livez" is whether the app is running (for autohealing) "readyz" is whether it's useful from customer point of view (e.g. for SLO) In the ideal world, those should be two separate handlers.
In addition to doubling the timeout we can also consider halving the probing period (from default 10s to 5s) and doubling failure thresholds (from 3 failed probes in row to 6). It would decrease the probability of etcd being restarted in case of temporarily overloaded master.
There is no option to keep using "/health" endpoint and changing timeout to something higher, as this is hardcoded in etcd: Also I think we should stop restarting etcd on NOSPACE or CORRUPT alarms. This way we will stop restarting on alarms and have ability to control timeout (and setting it to some higher value). Once etcd starts exposing some better endpoint that can be used for liveness probes, we will switch to that. WDYT?
/cc /triage accepted


Question: "//pkg/kubelet/apis/cri/testing:go_default_library", "//pkg/kubelet/remote/fake:go_default_library", "//vendor/github.com/stretchr/testify/assert:go_default_library", <ins> "//vendor/github.com/stretchr/testify/require:go_default_library", </ins> ], )

A race condition between fakeremote.Start() is spawned as a goroutine to listen for connections, and its caller assumes that the listener will immediately be ready to accept connections. This leads to intermittent test failures if the caller attempts before the fake remote runtime is ready to listen. The code needs to be refactored so that the fake remote returns from listen, and then spawns the server as a goroutine and returns. This can be reproduced by running the remoteruntimetest in a loop; it fails periodically. Failure can be accelerated by putting a time.Sleep() in fake.Start() immediately before the call to util.CreateListener; the failure is then effectively deterministic. Fix tested by running about 150,000 iterations without observing any failures.
/sig pod
Reference initial issue:
/sig node


Question: } if !nodeExist { <del> glog.Warningf( "Node %q does not exist. DiskIsAttached will assume vmdk %q is not attached to it.", vSphereInstance, volPath) return false, nil </del> <ins> glog.Errorf("DiskIsAttached failed to determine whether disk %q is still attached: node %q does not exist", volPath, vSphereInstance) return false, fmt.Errorf("DiskIsAttached failed to determine whether disk %q is still attached: node %q does not exist", volPath, vSphereInstance) </ins> } // Get VM device list

When a vm is reported as no longer present in cloud provider and is deleted by node controller, there are no attempts to detach respective volumes. For example, if a VM is powered off or paused, and pods are migrated to other nodes. In the case of vSphere, the VM cannot be started again because the VM still holds mount points to volumes that are now mounted to other VMs. In order to re-join this node again, you will have to manually detach these volumes from the powered off vm before starting it.
Once the node is powered off, the controller-manager doesn't send a detachVolume request to vSphere cloud provider. The controller-manager couldn't send the detachVolume request because it checks until the volume is unmounted on the node for a particular timeout. You can see the code over here - Since the volume is never unmounted, the controller manager skips the detachVolume request and issues the attachVolume() on another successfully. I think the controller-manager needs send a detachVolume() request even if the volume is mounted when the node is powered OFF. This is serious problem in our case (vSphere) as volume is now attached on another node and the node which is powered OFF cannot be powered ON again, as the node tries to acquire a lock on the volume to mount it on this node during the power ON. The acquire lock fails as it is already attached/mounted on other node. Also there are a few workarounds for this. When the Node is powered OFF, the cloud provider can detach the volume in NodeExists() - This method is called in DisksAreAttached() method. However i don't think this is a viable solution. : Can you provide your inputs on this. CCing
CC If a node goes offline (for any reason including being powered off) the attach/detach controller will eventually detach the volume even if the node never reports it as safely unmounted. Once a node becomes inaccessible, the node controller waits 5 min 40 seconds, before deleting all the pods that were scheduled to that node. When a pod referencing an attachable volume on that node is deleted, the attach/detach controller tries to detach the volume, but if the node has not reported the volume as safely unmounted (which would be the case if the node was powered off), the attach/detach controller waits 6 minutes before triggering detach anyway.

Once the volume is detached, any attempts to attach it to some other node will finally succeed. This means that it can take up to 15 minutes for this situation to rectify itself. Can you verify what amount of time you waited?
So do you mean that the volume can be attached to another node only after the volume is detached on the node which is powered OFF. In my case, i have waited for significant amount of time. I am sure it is more than 15 mins. And I see the volume succesfully attached to the new node but still not detached from the powered OFF node. In this case I cannot restart the node as explained above. Only if I delete the attached nodes manually I can power ON the node again.
If the storage system supports multi node attach for the access mode, then the attach to the new node should succeed immediately. If the storage systems supports only single node attach for the access mode then the attach calls to the new node fail fail until it is detached from the first node (this is the case with GCE PDs). If you are never seeing the volume detach from the original (powered off) node (even after 15 minutes), that sounds like a bug. AFAIK, this case is working fine in the GCE PD case ( tested this recently). If you have a consistent repro, to help debug, could you grab and share your controller-manager logs showing the original attach, subsequent attach, and lack of detach from the first node.
Sure. I will attach the logs. Between in our case we only support single node attach for a volume.
Between I am testing it on v1.4.7. Is this a change which is recently made to kubernetes code base to address this issue and not present in v1.4.7?
Please verify the logs from controller-manager. I see there is detach volume request being issued. I almost waited for 45 mins, but I didn't send any request attempting to detach volume as in - As in , since it failed to update node status, it couldn't make any further request for detachVolume(). Also, from the logs I see it is the kubernetes control plane which decides whether the storage plane is multi node attach or single node attach by attempting to send an attachVolume request on the new node.

And its not the cloud provider which says it to kubernetes control plane what type of storage plane it is. Correct me if I am wrong on this. : Can you please verify this.
That log is level 5. If your cluster is set up with log level 4, you will not see that log. When a node is down, the following events will happen with code release 4 or before. Once a node becomes inaccessible, the node controller waits for about 5 mins, and then deletes all the pods that were scheduled to that node. Once pods are deleted, Volume manager's reconciler will try to detach the volumes that belong to the pods. However, it will fail to issue to request first because it thinks the volumes are still mounted to the nodes according to the node status. (node status can be updated by the kubelet but since the node is down, the status is not updated). the reconciler fails to issue the detach request for a periods time, it times out (currently it is set to 6 minutes) and will issue detach request anyway and it should find that the volume is already detached since node is already terminated. So it will probably take more than 10 mins for reconciler marks the volume is detached to the node. We recently have a PR that change the above behavior during node termination because this PR adds a periodic check that whether the volumes are still attached. When node is terminated, the reconciler will very quickly (in a few seconds) find out the node is terminated and volume is detached and then mark this change in the actual state. {"log":"W1223 00:27:26. 1 ] Node "node2" does not exist. DisksAreAttached will assume vmdk [[vsanDatastore1] kubevols/redis-master] are not attached to it.n","stream":"stderr","time":"2016-12-23T00:27:26.429562004Z"} {"log":"I1223 00:27:26.

1 ] VolumesAreAttached: check volume "[vsanDatastore1] kubevols/redis-master" (specName: "redis-master-data") is no longer attachedn","stream":"stderr","time":"2016-12-23T00:27:26.44262004Z"} Please let me know if you have any questions about this part. Can you please verify this.
: I am using v1.4.7. Can you let me know if this holds true for v1.4.7? I don't see any behavior as mentioned by in the logs. Also you can see there is no detach call made on the node. As per this - if fails to update the node status after various attempts and hence detach call doesn't pass through. Also, from the logs I see it is the kubernetes control plane which decides whether the storage plane is multi node attach or single node attach by attempting to send an attachVolume request on the new node. And its not the cloud provider which says it to kubernetes control plane what type of storage plane it is. Also can you comment on the above question?
+1 on this. Same goes for k8s v1.5.2. Detach never happens when powering off a node with PVs attached. Disks has to be manually detached before being able to turn on the node again.
The reason detach never happens is as follows, is terminated controller periodically check whether volumes are still attached by sending requests to cloud provider. The function DisksAreAttached will check whether the node still exist. If not, it will assume the volume is no longer attached. If node does exist, check the information returned by the cloud provider to see whether volume is still attached. In this step, I believe it returns as volume is detached volume is already detached. So explicit detach will not be issued since step 2 and 3 already marks it as detached. I will check the function to see if there is potential problem in checking volume detached or not.
Cheers then I understand. I will also have a look.
is a workaround but ideally k8s should call detach for every attach that was issued, if the disk is already detached it should be a no-op. The ideal behavior should be that for each attach there will be a detach (which could be redundant).

The assumption in the current model seems to be that as long as the volume provider does not consider the disk attached the detach does not need to be issued which ignores the configuration for the VM. If the VM still considers the disk attached it should be reconfigured as well. The pod in this case is fine after being rescheduled to a new node but the VM is left with an invalid configuration.
Ok! The workaround seems legit for the time being. Thanks!


Question: package fs import ( <del> "bytes" </del> "fmt" <del> "os/exec" "strings" </del> <ins> "os" "path/filepath" "syscall" </ins> "golang.org/x/sys/unix" <del> "k8s.io/apimachinery/pkg/api/resource" </del> "k8s.io/kubernetes/pkg/volume/util/fsquota" ) <ins> type UsageInfo struct { Bytes  int64 Inodes int64 } </ins> // Info linux returns (available bytes, byte capacity, byte usage, total inodes, inodes free, inode usage, error) // for the filesystem that path resides upon. func Info(path string) (int64, int64, int64, int64, int64, int64, error) {

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: in calculates number of inodes wrong when there are hard links. What you expected to happen: inodes with hardlinks should only be counted once. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Issue is still not solved. There is an open PR waiting for review. /remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
still waiting for to be merged

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kubelet fails to run pods on Alpine Linux with busybox find. What you expected to happen: I expected the call to exit with success, not failure. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing GNU findutils, so busybox implementation of is used. Anything else we need to know?: failed because the argument is not implemented in busybox . The option is not specified in so it can not be expected to be supported in all implementations. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related bug Others: I think it would be better to use instead of Related to
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This is solved by which is waiting for a review. /remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Still waiting for the PR to be merged. /remove-lifecycle rotten
/remove-lifecycle rotten

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Running test suite on a cluster with workers on Alpine Linux, which uses busybox instead of GNU coreutils fails. The log shows that it fails to execute the command. What you expected to happen: I expected the call to exit with success. How to reproduce it (as minimally and precisely as possible): Install and run kublet on Alpine Linux without installing . Anything else we need to know?: It failed due to the option which is not implemented in busybox. The option is not specified in so I don't think it is reasonable to expect it to implemented in busybox. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: QEMU/KVM virtual machine OS (e.g: ):Kernel (e.g. ):Install tools: custom Network plugin and version (if this is a network-related bug): not network-related Others:
/sig Storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale fixes this
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
/remove-lifecycle rotten
Still waiting for PR to be merged


Question: import ( "testing" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/kubernetes/pkg/apis/core"

What happened: A pod that has init container with request for cpu and memory, and containers that make no request for cpu or memory reports a BestEffort QoS class. This is incorrect. What you expected to happen: Containers that make requests independent of lifecycle phase should impact quality of service calculation. Found in 1.12, looks to still be present in 1.14 dev. /cc
/priority important-soon
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/lifecycle frozen
I think this doesn't work very well with the current cpu manager logic. The problem is that a pod can't now have QoS class Guaranteed if it has init containers which don't have matching requests/limits. This means that normal containers in such a pod don't then get dedicated cpus even though the init containers are long gone when the app containers (with matching requests/limits) are finally started. This is quite non-intuitive. I would propose separating at least Guaranteed class calculation so that the init container resource requests wouldn't affect the calculation. What do you think?


Question: kube::test::get_object_assert 'pods b -n nsb' "{{${id_field:?}}}" 'b' # apply --prune must prune a kubectl apply --prune --all -f hack/testdata/prune/b.yaml <del> # check wrong pod doesn't exist output_message=$(! kubectl get pods a -n nsb 2>&1 "${kube_flags[@]:?}") kube::test::if_has_string "${output_message}" 'pods "a" not found' # check right pod exists kube::test::get_object_assert 'pods b -n nsb' "{{${id_field:?}}}" 'b' </del> <ins> # check wrong pod doesn't exist and right pod exists kube::test::wait_object_assert 'pods -n nsb' "{{range.items}}{{${id_field:?}}}:{{end}}" 'b:' </ins> # cleanup kubectl delete ns nsb

<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: Which test(s) are flaking: Testgrid link: Reason for failure: Anything else we need to know: /sig cli /cc /milestone v1.20 /prority important-soon
after creating the CRD, before applying the CR, kubectl should wait until includes :
/assign
/assign is this issue related to ?
no, this is simply kubectl trying to apply a CR before the server has made the API available at all
Got it! and I have filed to address this issue.
Still happening:
still happening: looks like the root cause is not related to CRD: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: /reopen
Reopened this issue. <detailsIn response to : still happening: I think the message Error from server (NotFound): "myobj" not found is expected, as the CR was not actually created: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, the actual error is later: Line 177 succeeds, and the next line fails:


Question: } func (rdm *resourceDiscoveryManager) addGroupVersionLocked(source Source, groupName string, value apidiscoveryv2beta1.APIVersionDiscovery) { <del> klog.Infof("Adding GroupVersion %s %s to ResourceManager", groupName, value.Version) </del> if rdm.apiGroups == nil { rdm.apiGroups = make(map[groupKey]*apidiscoveryv2beta1.APIGroupDiscovery)

I upgraded k3s from to and noticed a log message is repeated in the I also see this message when deploying a container that uses apiserver aggregation, such as in other distros such as in the api-server log. The message should not repeated in the logs. This causes a nuisance and the logs eventually are rotated. Upgrade kubernetes to and install a container that uses uses apiserver aggregation such as or . For reference the original issue was reported here: <details</details<details</details<details</details<details</details<details</details<details</details
/cc /sig api-machinery The log was in since v1.26.
Thanks for raising this, opened as a potential fix.
/triage accepted
/assign


Question: f: f, } It("should test privileged pod", func() { <del> hostExecPod := NewHostExecPodSpec(f.Namespace.Name, "hostexec") pod, err := config.getPodClient().Create(hostExecPod) expectNoError(err) config.hostExecPod = pod </del> <ins> config.hostExecPod = LaunchHostExecPod(config.f.Client, config.f.Namespace.Name, "hostexec") </ins> By("Creating a privileged pod") config.createPrivilegedPod()

Test failed with the following error: Expected error: <*errors.errorString | 0xc208509700: { s: "Error running &{/jenkins-master-data/jobs/kubernetes-e2e-gce-parallel/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.197.78.129 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-e2e-gce- exec --namespace=e2e-tests-e2e-privilegedpod-lviv7 hostexec -- /bin/sh -c curl -q 'http://10.245.1.19:8080/shell?shellCommand=ip+link+add+dummy1+type+dummy'] [] error: pod hostexec is not running and cannot execute commands; current phase is Pendingn [] 0xc2085241c0 exit status 1 true [0xc208418828 0xc208418848 0xc208418868] [0xc208418828 0xc208418848 0xc208418868] [0xc208418840 0xc208418860] [0x7f0740 0x7f0740] 0xc208340c60}:nCommand stdout:nnstderr:nerror: pod hostexec is not running and cannot execute commands; current phase is Pendingnn", } Error running &{/jenkins-master-data/jobs/kubernetes-e2e-gce-parallel/workspace/kubernetes/platforms/linux/amd64/kubectl [kubectl --server=https://104.197.78.129 --kubeconfig=/var/lib/jenkins/jobs/kubernetes-e2e-gce- exec --namespace=e2e-tests-e2e-privilegedpod-lviv7 hostexec -- /bin/sh -c curl -q 'http://10.245.1.19:8080/shell?shellCommand=ip+link+add+dummy1+type+dummy'] [] error: pod hostexec is not running and cannot execute commands;

current phase is Pending [] 0xc2085241c0 exit status 1 true [0xc208418828 0xc208418848 0xc208418868] [0xc208418828 0xc208418848 0xc208418868] [0xc208418840 0xc208418860] [0x7f0740 0x7f0740] 0xc208340c60}: Command stdout: stderr: error: pod hostexec is not running and cannot execute commands; current phase is Pending not to have occurred
The example failure is here:
This happened again - increasing priority to P0 according to the policy we discussed: The failure is here:
FYI
Yet another example:
Can you point to the publicly visible artifacts for those tests? It's not obvious to me from the description why I was @ mentioned
- I though that you were involved in this test - if not, please ignore that I mentioned you
Another:
I think is the right person to tag.
It looks like the test doesn't wait for the hostexec pod to start running:
Easy fix, I'll take it.
Thanks Jeff.


Question: assert.NoError(t, err) assert.True(t, f.cpuHardcapping, "cpu hardcapping is expected to be enabled") } <ins> func TestGetCapacity(t *testing.T) { ephemeralStorageFromCapacity := int64(2000) ephemeralStorageFromCadvisor := int64(8000) mockCtrl := gomock.NewController(t) defer mockCtrl.Finish() mockCtrlError := gomock.NewController(t) defer mockCtrlError.Finish() mockCadvisor := cadvisortest.NewMockInterface(mockCtrl) rootfs := cadvisorapiv2.FsInfo{ Capacity: 8000, } mockCadvisor.EXPECT().RootFsInfo().Return(rootfs, nil) mockCadvisorError := cadvisortest.NewMockInterface(mockCtrlError) mockCadvisorError.EXPECT().RootFsInfo().Return(cadvisorapiv2.FsInfo{}, errors.New("Unable to get rootfs data from cAdvisor interface")) cases := []struct { name                                string cm                                  *containerManagerImpl expectedResourceQuantity            *resource.Quantity expectedNoEphemeralStorage          bool enableLocalStorageCapacityIsolation bool }{ { name: "capacity property has ephemeral-storage", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity: v1.ResourceList{ v1.ResourceEphemeralStorage: *resource.NewQuantity(ephemeralStorageFromCapacity, resource.BinarySI), }, }, expectedResourceQuantity:            resource.NewQuantity(ephemeralStorageFromCapacity, resource.BinarySI), expectedNoEphemeralStorage:          false, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity:          v1.ResourceList{}, }, expectedResourceQuantity:            resource.NewQuantity(ephemeralStorageFromCadvisor, resource.BinarySI), expectedNoEphemeralStorage:          false, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage, error from rootfs", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisorError, capacity:          v1.ResourceList{}, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: true, }, { name: "capacity property does not have ephemeral-storage, cadvisor interface is nil", cm: &containerManagerImpl{ cadvisorInterface: nil, capacity:          v1.ResourceList{}, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: true, }, { name: "LocalStorageCapacityIsolation feature flag is disabled", cm: &containerManagerImpl{ cadvisorInterface: mockCadvisor, capacity: v1.ResourceList{ v1.ResourceCPU:    resource.MustParse("4"), v1.ResourceMemory: resource.MustParse("16G"), }, }, expectedNoEphemeralStorage:          true, enableLocalStorageCapacityIsolation: false, }, } for _, c := range cases { t.Run(c.name, func(t *testing.T) { defer featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, kubefeatures.LocalStorageCapacityIsolation, c.enableLocalStorageCapacityIsolation)() ret := c.cm.GetCapacity() if v, exists := ret[v1.ResourceEphemeralStorage]; !exists { if !c.expectedNoEphemeralStorage { t.Errorf("did not get any ephemeral storage data") } } else { if v.Value() != c.expectedResourceQuantity.Value() { t.Errorf("got unexpected %s value, expected %d, got %d", v1.ResourceEphemeralStorage, c.expectedResourceQuantity.Value(), v.Value()) } } }) } } </ins>

When trying to join a second or sometimes third control plane node, fails with: something that looks like an etcd hangup ("Initial timeout of 40s passed") CrashLoopBackOff for one of the kube-apiserver pods The k8s suggest opening an issue here. Successful join op Here goes... First to init the first control plane node (1 of 3 in an HA cluster). This is successful. Contents of : Apply CNI plugin: Looks fine at this point. shows all Running. Next join 2nd control plane node: This seems to go off without a hitch also. The new node comes up as Ready and all new pods Running. Contents of here: Now for the third, failing node: same exact join command, same exact config. It is really a replica node of the 2nd control plane node, and yet times out. Output: Then going back to initial control node. Output of : Uh-oh. So CrashLoopBackOff shows up and one etcd pod is missing completely. No. Kubernetes version (use ): Cloud provider or hardware configuration: AWS EC2 OS (e.g: ): Ubuntu 18.04.5 LTS Kernel (e.g. ): Network plugin and version (if this is a network-related bug): weave-net
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cluster-lifecycle
/sig usability
Can you check etcd yaml in node in ? Also check the log on node to know why the etcd pod is not created. A known issue is for ephemeral-storage check, you can remove the ephemeral storage requests in and see if the etcd pod can be created.
Okay, I re-ran this exact setup on a newly provisioned set of nodes and ran into the same error.

xx-xx-xx-248: initial control plane node (ok) xx-xx-xx-131: second control plane node (ok) xx-xx-xx-176: third control plane node (fails to join) Also want to add the comment that I am joining the nodes in serial rather than in parallel, since I have in other issues seen parallel joins described as an issue for etcd. With this latest run I ran with and fetched some additional log info as referenced by Contents of on the failing node: Output from the failed on the failing node: is same as original post. I am -ing the output to and guessing I might need to capture stderr too. But that's outside the scope of this issue. And lastly kubelet log on the failing node (from ): As shown above this is littered wth (newlines for readability): Final note: the IP changes in this syslog from .65 because the nodes are rebooted before any init or join operations are ran.
/sig node storage
Going to give a try. In my situation here, each node is 65G disk and 4G RAM with nothing else really going on on them besides the k8s initialization.
It looks like applying as outlined in provided a band-aid and the final control plane node joined successfully. The param is successfully removed from . For other kubeadm users looking to apply this fix, the for is: The key is that the argument is a directory, not the patch file specifically. In my case, the file sat at and the invocation is then
I've updated PR to reflect that it will fix this issue Also, note that was merged yesterday, and will be cherry-picked into future 1.20 and 1.21 releases.
thanks for checking the logs. let's keep a single tracking issue for this problem: /close
Closing this issue. <detailsIn response to : Jun 09 11:49:50 ip-xx-xx-xx-176 kubelet[4641]: I0609 11:49:50.

4641 ] "Failed to admit pod, unexpected error while attempting to recover from admission failure" pod="kube-system/etcd-ip-xx-xx-xx-176" err="preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ]" Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

on the new node fails sometimes. The following error appears in the log: on the new node succeeds every time Join a control plane node The node has over 40 GB of free space on its root partition. I some more logging to the following functions: , , , , , , It appears that the ephemeral storage is detected a bit too late: /sig node storage Kubernetes version (use ): 1.20.4, commit Kernel (e.g. ): 4.9.252-1.ph2
/assign I will try to reproduce it today.
LocalStorageCapacityIsolation is a feature gate for kubelet. Can you check if you disable it? ReadyCondition will check ephemeral-storage as required capacities if this feature gate is enabled in kubelet configuration. Would you check it first?
[x] 1.8 alpha ephemeral-storage [x] 1.10 Promote LocalStorageCapacityIsolation feature to beta [x] 1.15 KEPs Quotas for ephemeral-storage for LocalStorageCapacityIsolationFSQuotaMonitoring default to false is Alpha since 1.15 [x] Failed to promote LocalStorageCapacityIsolationFSQuotaMonitoring feature to beta [x] 1.20 Honor disabled LocalStorageCapacityIsolation in scheduling I am no sure whether there is a plan to GA LocalStorageCapacityIsolation?
I tried to fix it in . Need some testing later.
It appears that the pod fails before sees the storage capacity:
Would you try this ? I build it with on PR branch.
I tried to apply the PR to 1.20.4. After four successful s, one failed. I got the following error:
! It means that during the node resource updated the first time, the cadvisor is not available yet. If we wait here, it would be a problem. I would think more of how to fix it.
Hi, Same problem here. I'm using K8s v1.20.4 with CentOS7 (2 cores, 4GB and 20GB storage) Cluster creation with kubeadm init always works, but when I try to add several master nodes with kubeadm join it takes a lot of time waiting for etcd pod to be created. In most of cases, etcd pod never runs.

The rest of components (APIserver, controller, scheduler and proxy) exists and works (except APIServer which reboots trying to connect the missing etcd) Checking the kubelet logs in the missing etcd hosts I have found this: Mar 10 14:23:27 kmaster-02 kubelet[9448]: W0310 14:23:27. 9448 ] Failed to admit pod etcd-kmaster-02_kube-system() - Unexpected error while attempting to recover from admission failure: preemption: error finding a set of pods to preempt: no set of running pods found to reclaim resources: [(res: ephemeral-storage, q: ), ] That node have 17GB storage free. If kubelet is restarted, the etcd (pod and container in docker ps -a) appears. If I could help doing something please tell me. I would like to help
For anyone who is experiencing this issue and still wants to use kubeadm until it is fixed: You can remove the ephemeral-storage request from the etcd static pod and the join will work. Just make sure that your nodes actually have more than 100MB free storage left. You can reinsert the storage request by patching the etcd manifests after the nodes are up. To patch etcd during a create the following file (the filename matters, the file location doesn't) and use to join additional control plane nodes.
Thanks for the workaround I'm encountering this same issue and the workaround worked for me. Edit: the patch I tried doesn't work though (not sure if this is really a problem). I issued the command: and the response shows that the resource can't be patched this way: There may be another way but I'm still fairly new to Kubernetes so it is not obvious to me at the moment. Anyhow, thanks for the suggestion!
I updated Add a start-delay of 10 seconds for a condition that the capacity of node ephemeral-storage is 0. Not sure if this will fix it.
we are starting to see reports about this every week.
cc
cc
From it sounds like the issue is kubeadm etcd ephemeral-storage requests are too big for some environments? So far this is sounding like a kubeadm issue, not a kubernetes issue.
it seems to be something else.

i doubt that users are trying machines that don't satisfy 100Mi. for example, the report in the OP points out that the machine has 40GB.
I can confirm that the VMs on which I installed k8s control plane components had sufficient disk space when I encountered the problem. I'm running Kubernetes v1.20.4 and kubeadm v1.20.4.
I think the root cause here sounds right. There is kind of race here. adding delay will help, but it does not have 100% guarantee. Not sure whether it is easy to add logic to make sure capacity is reported correctly before starting pods..
Did something change in 1.20 regarding the ordering of cadvisor and node startup?
If the reason would be a disk problem(or something we can figure out) and the delay can solve it, I think we should add a warning besides the delay.
We need to figure out what triggers the race condition. This only happens on some machines. I have not seen this myself.
We're able to repro this nearly 100% of the time with Cluster API Provider Azure + k8s 1.20.x + Ubuntu 18.04 + default template, which makes k8s 1.20 pretty much unusable at this point. Trying to see if this repros with other non-Azure cluster-api providers with kubeadm 1.20.
Does anyone know why this only repros on and not ? The ephemeral storage resource requirement here is used in common by both and : In practice I can workaround this issue in cluster-api + Azure by adding a sleep command to my spec, but it requires a significant number to make work ~100% of the time (like 90 seconds). Does the flow take significantly longer before it gets to the part where it submits an etcd pod to the apiserver? Is there any other explanation for this being a -only symptom?
i can see this failing for any static pod with ephemeral storage requests. that is, unless there is proof that this is somehow specific to the kubeadm static pod for etcd that kubeadm deploys.
I agree that the specific kubeadm-delivered etcd pod is not interesting. (In fact it's very permissive, it's only asking for 100Mi.)

What definitely seems to be happening (in Azure) is that kubelet takes a long time to know about its underlying node's ephemeral-storage availability. Kubelet starts: ~2 mins later, it complains that it can't schedule etcd: Again, why does this not repro on nodes created by ?
it also seems to be an even more scoped problem because it's not reproducible for me locally (i create a number of clusters every day, usually), the kubeadm CI or in other high level projects that use kubeadm (e.g. kind, minikube, etc). we should try to find a common reason / condition from all the reporting users that are seeing this.
Experiencing same problem on openSuSE Kubic/Tumbletweed snapshot with kubeadm/kubelet 1.20.2, kernel 5.11.15 running on Xen, 2 CPU, 4GB ram 30GB disk (BRTFS)
Problem consistently appear/disappear when switching between snapshots (Working) and , so a package update in that period has probably triggered this. Following files contains changes between those two snapshots.
are you suggesting a root cause in underlying Linux storage driver surface area that is adding delay/uncertainty to kubelet's ability to introspect available ephemeral-storage on the node host OS?
A bit of luck combined with 6th sense i got it narrowed down to systemd update here Looking at the changelog, one item sticks out: Anyone else enabled cgroupsv2 on i.e. the CentOS cases (it needs to be enabled explicitly for that AFAIK)?
ACK on your latest respond regarding unified cgroup hierarchy. Is there a quick, cross-distro way to verify that any folks who are repro'ing have a similar systemd config?
The precence of file/dir /sys/fs/cgroup/cgroup.controllers should mean you are using cgroupsv2 ( Ref: ) I can confirm that the file is present on the kubic snapshot which is not working. It is not present on the one that does work. ( I'm using cri-o as runtime which should support cgroupsv2 ) cgroup2 support was introduced for K8S in 1.19 ( ).
Thanks. So tl;dr your working theory is that cgroupsv2 + kubelet ephemeral-storage admission has some pathologies.

I'll post quickly with what my repro containerd + Ubuntu 18.04-LTS OS surface area looks like.
I ran into this issue consistently on Ubuntu 20.04.2 LTS using cri-o but is not present on my systems.
I can also confirm that cgroupsv2 is not a factor in my repro'ing environment:
Also, I'm not using cri-o, containerd here:
Ok - so much for that theory. But still: What I did to narrow it to systemd-update was taking the working kubic snapshot () and run zypper update systemd which updates systemd (along with systemd-sysvinit and udev packages to same version). That consistently made it fail. So I think a pointer to the root-cause might still be hiding somewhere around this changelog - even if it is not cgroupsv2. That part was just a theory (seemed like a "big" thing), but the update of systemd package is fairly hard evidence.
I have identified where these particular symptoms (probably) originate, and cursory testing suggests something like this eliminates the issue: I'll be advocating for a few mins in today's sig-node discussion if any folks following this thread want to chime in. cc
/sig cluster-lifecycle /triage accepted
I've also observed that on some occasions the node allocatable storage is in a continual state of reporting "0". Restarting kubelet immediately fixed that. More debugging to do....
did you manage to reproduce this with an arbitrary static pod that is not the etcd one?
I was not able to reproduce in a non-kubeadm test, but I suspect that the way kubelet bootstrap is implemented there (and possibly environmental reasons) the static pod handling was always happening after this code executed: (the above is in the containerManager method) kubeadm w/ ephemeral-storage requirements in etcd is the most effective repro at this time
the writing of the etcd pod manifest in during is after the kubelet has already started and i wonder if that is contributing to the observed failures. probably not given this is a cadvisor related problem. to verify that i would write an arbitrary static pod manifest with the same request in : before (which also starts the kubelet) right after the kubelet has started if is causing such racy problems i don't think it should be graduated to GA.

though, it has been Beta for ~12 releases (on by default), so it's already used everywhere.
Based on your assessment do you think we should remove the ephemeral-storage requirement from default kubeadm etcd static pod? I can make that change.
based on the investigations here (thanks all), it feels like is a feature that may or may not work (for static pods at least). it doesn't feel right to mitigate a kubelet bug on the side of the deployer, but given how many reports there are about this i think it should be done at this point. happy to LGTM approve your PRs. note it must also be backported to all versions in support (back to 1.19). suggesting the following release note: we should keep this issue open, too.
Sounds good. I'll submit a PR soon with the kubeadm change, and I'll continue to drive toward a root cause solution on the kubelet side. I'll bring this up at sig-node tomorrow. Thanks!
is a possible duplicate of this issue - seeing a CrashLoopBackOff failure for kube-apiserver under a similar scenario to those mentioned here, e.g. failure to join a third control plane node with mentions of in the kubelet logs.
An ugly as sin workaroud: (please don't hate me :D )
It seems like this could be closed with and related cherry picks now being part of the newest releases? Bumping to 1.21.2 allowed me to remove the hack for etcd and several test deployments have steered clear entirely of the original issue. Thanks
great news. I'd prefer to pin this issue to this PR: That PR will deliver more race condition resilience for all static pod + ephemeral-storage requirement scenarios.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at .

/lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:


Question: "$ref": "#/components/schemas/io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1.WebhookConversion" } ], <del> "description": "webhook describes how to call the conversion webhook. Required when `strategy` is set to `Webhook`." </del> <ins> "description": "webhook describes how to call the conversion webhook. Required when `strategy` is set to `"Webhook"`." </ins> } }, "required": [

Currently we have inconsistency in most of the API where the field name should be used in description so the generated docs match the API fields people actually set. A linter could be for checking To avoid confusing users
/good-first-issue
/triage accepted
/assign
Hey can you provide some more information for this issue
are you still working on this?
Hey feel free to take it. I am unassigning myself
/triage needs-information /remove-good-first-issue
/sig api-machinery


Question: <ins> /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package cm import ( "k8s.io/api/core/v1" evictionapi "k8s.io/kubernetes/pkg/kubelet/eviction/api" ) // hardEvictionReservation returns a resourcelist that includes reservation of resources based on hard eviction thresholds. func hardEvictionReservation(thresholds []evictionapi.Threshold, capacity v1.ResourceList) v1.ResourceList { if len(thresholds) == 0 { return nil } ret := v1.ResourceList{} for _, threshold := range thresholds { if threshold.Operator != evictionapi.OpLessThan { continue } switch threshold.Signal { case evictionapi.SignalMemoryAvailable: memoryCapacity := capacity[v1.ResourceMemory] value := evictionapi.GetThresholdQuantity(threshold.Value, &memoryCapacity) ret[v1.ResourceMemory] = *value case evictionapi.SignalNodeFsAvailable: storageCapacity := capacity[v1.ResourceEphemeralStorage] value := evictionapi.GetThresholdQuantity(threshold.Value, &storageCapacity) ret[v1.ResourceEphemeralStorage] = *value } } return ret } </ins>

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: After running some tests, we're seeing that the combined usage of kubelet + docker can be around 2Gi. We need to be able to account for this in scheduling to avoid crashing these services. The first step towards this is using kube-reserved and system-reserved to avoid oversubscribing the virtual memory space available on the Windows nodes with too many containers. What you expected to happen: It should reduce node allocatable memory by kube-reserved + system-reserved. How to reproduce it (as minimally and precisely as possible): This logs an error in the kubelet log such as: code: Anything else we need to know?: /milestone v1.13 /assign feiskyer /sig windows This blocks a PR where I want to set it by default to 2Gi Environment: Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine
You must be a member of the github team to set the milestone. <detailsIn response to : Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: This logs an error in the kubelet log such as: code: /milestone v1.13 /assign feiskyer Kubernetes version (use ): v1.11.3 (also v1.12) Cloud provider or hardware configuration: Azure OS (e.g. from /etc/os-release): Windows Server version 1803 Install tools: acs-engine Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.13
/priority important-soon


Question: exit 0 fi <ins> # Ensure we have the right godep version available kube::util::ensure_godep_version </ins> if [[ -z ${TMP_GOPATH:-} ]]; then # Create a nice clean place to put our new godeps _tmpdir="$(mktemp -d -t gopath.XXXXXX)"

Didn't see an issue for this. Godep recently updated its version to v80. This broke several builds that directly godep, grabbing latest. /kind bug cc Example https://k8s- Here are some examples where we grab master:
We have helpers to grab godep. We should use them everywhere.
/cc
/assign


Question: PROXY_PORT= } <del> # Starts "kubect proxy" to test the client proxy. You may pass options, e.g. # --api-prefix. </del> <ins> # Starts "kubect proxy" to test the client proxy. $1: api_prefix </ins> function start-proxy() { stop-proxy

Now that we've moved the experimental api to /apis/experimental... it's inaccessible from by default. We should fix that. who stepped on this earlier
security questions for you: this prefix was apparently in response to a security review quite some time ago. I don't understand how this adds much security. Can we just remove it? If not, is it acceptable to change the default from "/api/" to "/api", which would match both api locations? Otherwise we'll have to take a comma-separated list or something...
Ref
Oh, so this is not related to the security requests! And the original use case was just to re-root the URL path so it didn't interfere with static content. Sorry, I got confused because the last time I monkeyed with that, it was in conjunction with the allow/deny list that actually was the security request. In that case, let's see if we can change the default to "/", if that will break anything.
That change got introduced as part of to protect against XSRF in kubectl proxy. But I think the more important part of that change was that it disallows POST/PUT/PATCH by default. So as long as there is no state changing action outside of /api/ that can be triggered with a GET it should be safe to remove.


Question: } } <del> // Although the CDI devices are expected to be empty when this feature is disabled, we still // guard this with a feature gate to avoid any potential issues. if utilfeature.DefaultFeatureGate.Enabled(kubefeatures.DevicePluginCDIDevices) { // We construct a resource ID from the pod UID and container name. // This ID has no semantic meaning, and is only used to ensure that the generated CDI annotation key is unique // for a given container. Since this is only called once per pod-container combination, this should be the case. resourceID := podUID + "-" + contName cdiAnnotations := getCDIAnnotations(resourceID, allCDIDevices, annotationsMap) opts.Annotations = append(opts.Annotations, cdiAnnotations...) } </del> return opts } <del> // getCDIAnnotations returns the cdi annotations for a given container. // This creates a CDI annotation with a key of the form: devicemanager_{{resourceID}}. // The value of the annotation is a comma separated list of sorted CDI device IDs. // If the annotation key is already defined in the provided annotations map, then the existing value is used. func getCDIAnnotations(resourceID string, cdiDevices sets.Set[string], annotationsMap map[string]string) []kubecontainer.Annotation { // We sort the CDI devices to ensure that the annotation value is deterministic. sortedCDIDevices := sets.List[string](cdiDevices) annotations, err := cdi.GenerateAnnotations(types.UID(resourceID), "devicemanager", sortedCDIDevices) if err != nil { klog.ErrorS(err, "Failed to create CDI annotations") return nil } var cdiAnnotations []kubecontainer.Annotation for _, annotation := range annotations { if e, ok := annotationsMap[annotation.Name]; ok { klog.V(4).InfoS("Skip existing annotation", "annotationKey", annotation.Name, "annotationValue", annotation.Value) if e != annotation.Value { klog.ErrorS(nil, "Annotation has conflicting setting", "annotationKey", annotation.Name, "expected", e, "got", annotation.Value) } continue } klog.V(4).InfoS("Add annotation", "annotationKey", annotation.Name, "annotationValue", annotation.Value) annotationsMap[annotation.Name] = annotation.Value cdiAnnotations = append(cdiAnnotations, kubecontainer.Annotation{Name: annotation.Name, Value: annotation.Value}) } return cdiAnnotations } </del> // getCDIDeviceInfo returns CDI devices from an allocate response func getCDIDeviceInfo(resp *pluginapi.ContainerAllocateResponse, knownCDIDevices sets.Set[string]) []kubecontainer.CDIDevice { var cdiDevices []kubecontainer.CDIDevice

Kubelet should stop using annotations to pass CDI device IDs to CRI runtimes. CDIDevices CRI field should be used for this purpose. This should be done when Kubelet that doesn't support CRI field reaches EOL and two major CRI runtimes support CRI field. Here is a PR comment that gives more details about this: Kubelet started using CDIDevices CRI field in . Previous version 1.27 will reach EOL .CRI-O started using it in . Previous version 1.27 will reach EOL together with Kubernetes 1.27.Containerd started using it in . Version 1.7 will reach EOL in a 6 months after Containerd 2.0 release Previous version 1.6 doesn't support CDI injections and should not be considered. So, safer option would be to wait until Containerd 1.7 reaches EOL. However it's relatively safe to do it earlier when 1.7.0 and 1.7.1 have little chances to be in real use. Containerd v1.7.0 and v1.7.1 is not used by the Linux distributions from , so it's safe to remove annotations support from the Kubelet in 1.31 in my opinion.
/sig node
/triage accepted /priority important-longterm
/assign


Question: kube-controller-manager kube-scheduler kube-proxy <ins> kube-log-runner </ins> kubeadm kubectl )

/kind feature Hey, in K8s 1.23 we are planning to deprecate multiple klog flags. Deprecated flags: , , , , , , , , , , , One of the alpha milestones is requires us to remove usage of deprecated flags within K8s repo. Examples found for windows [ ] [ ] /sig windows /wg structured-logging
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign I've started looking into this. Part of the problem is that go-runner is not part of kubernetes-node-windows- I suspect that is where the scripts get and kube- from.


Question: "sync" "time" <ins> "k8s.io/apimachinery/pkg/util/wait" "github.com/onsi/ginkgo" "github.com/onsi/gomega" </ins> v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/labels"

Test: [sig-node] Density [Serial] [Slow] create a sequence of pods latency/resource should be within limit when create 10 pods with 50 background pods /kind flake <!-- If this is a flake, please add: /kind flake --/sig node
/triage accepted /priority important-soon
Although the pod polling exhausted 4 times (interval 2s, total 6s+) to get pod Running, but from the kubelet log, the time from the pod found to Running status reported is almost less than 5s, so wo may reduce the poll interval to 1s, or make this time more relaxed
http://perf- ! This is caused by as . The commit will be reverted, and it make fail yesterday. ! EDITED I mean if this flake points out there are some other performance downgrades in v1.24. Only recently, no version 99% -- -- v1.20 ~2200ms v1.21 ~2200ms v1.22 ~2300ms v1.23 ~2650ms v1.24(before) ~2200ms v1.24(after ) ~6600ms
Editing title as this was also failing on containerd: /milestone v1.24 I think this confirms that the revert above fixed the issue, as the failures stopped after the revert merged. /close
Closing this issue. <detailsIn response to : Editing title as this was also failing on containerd: Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: <ins> /* Copyright 2015 Google Inc. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package e2e import ( "fmt" "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/labels" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/wait" . "github.com/onsi/ginkgo" . "github.com/onsi/gomega" ) // Delete a Replication Controller and all pods it spawned func DeleteRC(c *client.Client, ns, name string) error { rc, err := c.ReplicationControllers(ns).Get(name) if err != nil { return fmt.Errorf("Failed to find replication controller %s in namespace %s: %v", name, ns, err) } rc.Spec.Replicas = 0 if _, err := c.ReplicationControllers(ns).Update(rc); err != nil { return fmt.Errorf("Failed to resize replication controller %s to zero: %v", name, err) } if err := wait.Poll(time.Second, time.Minute*20, client.ControllerHasDesiredReplicas(c, rc)); err != nil { return fmt.Errorf("Error waiting for replication controller %s replicas to reach 0: %v", name, err) } // Delete the replication controller. if err := c.ReplicationControllers(ns).Delete(name); err != nil { return fmt.Errorf("Failed to delete replication controller %s: %v", name, err) } return nil } // Launch a Replication Controller and wait for all pods it spawns // to become running func RunRC(c *client.Client, name string, ns, image string, replicas int) { defer GinkgoRecover() var last int current := 0 same := 0 defer func() { By("Cleaning up the replication controller") err := DeleteRC(c, ns, name) Expect(err).NotTo(HaveOccurred()) }() By(fmt.Sprintf("Creating replication controller %s", name)) _, err := c.ReplicationControllers(ns).Create(&api.ReplicationController{ ObjectMeta: api.ObjectMeta{ Name: name, }, Spec: api.ReplicationControllerSpec{ Replicas: replicas, Selector: map[string]string{ "name": name, }, Template: &api.PodTemplateSpec{ ObjectMeta: api.ObjectMeta{ Labels: map[string]string{"name": name}, }, Spec: api.PodSpec{ Containers: []api.Container{ { Name:  name, Image: image, Ports: []api.ContainerPort{{ContainerPort: 80}}, }, }, }, }, }, }) Expect(err).NotTo(HaveOccurred()) By(fmt.Sprintf("Making sure all %d replicas exist", replicas)) label := labels.SelectorFromSet(labels.Set(map[string]string{"name": name})) pods, err := c.Pods(ns).List(label) Expect(err).NotTo(HaveOccurred()) current = len(pods.Items) failCount := 5 for same < failCount && current < replicas { Logf("Controller %s: Found %d pods out of %d", name, current, replicas) if last < current { same = 0 } else if last == current { same++ } else if current < last { Failf("Controller %s: Number of submitted pods dropped from %d to %d", last, current) } if same >= failCount { Logf("No pods submitted for the last %d checks", failCount) } last = current time.Sleep(5 * time.Second) pods, err = c.Pods(ns).List(label) Expect(err).NotTo(HaveOccurred()) current = len(pods.Items) } Expect(current).To(Equal(replicas)) Logf("Controller %s: Found %d pods out of %d", name, current, replicas) By("Waiting for each pod to be running") same = 0 last = 0 failCount = 6 unknown := 0 pending := 0 current = 0 for same < failCount && current < replicas { current = 0 pending = 0 unknown = 0 time.Sleep(10 * time.Second) for _, pod := range pods.Items { p, err := c.Pods(ns).Get(pod.Name) Expect(err).NotTo(HaveOccurred()) if p.Status.Phase == api.PodRunning { current++ } else if p.Status.Phase == api.PodPending { pending++ } else if p.Status.Phase == api.PodUnknown { unknown++ } } Logf("Pod States: %d running, %d pending, %d unknown ", current, pending, unknown) if last < current { same = 0 } else if last == current { same++ } else if current < last { Failf("Number of running pods dropped from %d to %d", last, current) } if same >= failCount { Logf("No pods started for the last %d checks", failCount) } last = current } Expect(current).To(Equal(replicas)) } // This test suite can take a long time to run, so by default it is disabled // by being marked as Pending.  To enable this suite, remove the P from the // front of PDescribe (PDescribe->Describe) and then all tests will // be available var _ = PDescribe("Density", func() { var c *client.Client var minionCount int var RCName string var ns string BeforeEach(func() { var err error c, err = loadClient() expectNoError(err) minions, err := c.Nodes().List() expectNoError(err) minionCount = len(minions.Items) Expect(minionCount).NotTo(BeZero()) ns = api.NamespaceDefault }) AfterEach(func() { // Remove any remaining pods from this test DeleteRC(c, ns, RCName) }) It("should allow starting 100 pods per node", func() { RCName = "my-hostname-density100-" + string(util.NewUUID()) RunRC(c, RCName, ns, "dockerfile/nginx", 100*minionCount) }) }) </ins>

What would you like to be : Remove example from . Why is this needed: While trying to get all unit tests to run locally, I tried using this and it doesn't work, at least not from the Kubernetes root. The reason it doesn't work is that it is not equivalent to , because some directories when it runs unit tests. On a side note, I think it is actually a little confusing presenting both and alternatives intermixed within the testing guide. I wonder if the examples would be better off consolidated in their own section toward the end of the guide.
/sig testing /kind documentation
/remove-kind feature
I can do this for my first contribution, maybe it's better if I specify in the Run all test paragraph : It's better to use make test because it excludes some directories when it runs unit tests. Do you thinks it's enough ?
I will defer to someone from the sig testing team to answer that question. definitely won't work, so that should be removed from the docs altogether, IMO. The other examples, I'm not sure if there is a reason to keep them in the docs or not. They can work sometimes, depending on what you are testing, but they don't work universally... for example, if I try to something under it will likely fail because at least some of those tests have some external dependencies and are not true "unit tests".
+1 (not a sig-testing member though)
do you mind elaborating on I just ran it from root and seems like most of the tests run for me, just due to GOPATH being unset I get a lot of If there were instructions on setting the GOPATH and we moved the go testing bit to it's own area of the guide, would that work?
sure. I probably should have provided more specific examples. The main problem is the directory. It seems the tests in that folder are not really unit tests because they require a KUBEADM_PATH variable to be set in order to run and will actually call the kubeadm executable. If I just do from the project root, it will try to run these tests and fail. On the other hand, if I use , I don't have a problem because it specifically excludes certain folders where the tests are not really unit tests. While may be fine to use within some subdirectories, I don't think it can be used at the project root.

I'd recommend opening a bug against those tests in particular
OK, I can do that. Makes sense to move them to integration probably, or remove the dependencies that cause them to fail.
Closing this issue in favor of fixing the problem in the tests. New issue is


Question: return nil } <ins> func testReachabilityOverExternalIP(externalIP string, sp v1.ServicePort, execPod *v1.Pod) error { return testEndpointReachability(externalIP, sp.Port, sp.Protocol, execPod) } </ins> func testReachabilityOverNodePorts(nodes *v1.NodeList, sp v1.ServicePort, pod *v1.Pod, clusterIP string) error { internalAddrs := e2enode.CollectAddresses(nodes, v1.NodeInternalIP) externalAddrs := e2enode.CollectAddresses(nodes, v1.NodeExternalIP)

What would you like to be : e2e test for service ExternalIPs Why is this needed: There is no test covering Service ExternalIPs Since we can not assume that the e2e binary will have connectivity against the nodes and that we can add routes to the environment, we are going to test it from a pod. I've verified that this works locally with the following commands: The test should leverage current framework, maybe something as A good example is here, maybe adding a ?
/sig network /help wanted /triage accepted
/help wanted /good-first-issue
/assign
I'd like to pick this up!
Ok,I'm not started :) /unassign /assign
There are two different externalIP codepaths, one for the case where the claimed external IP exists as an IP on the default interface, and one for where it does not. I suppose we could just temporarily add an IP to the default interface...
good idea Dan Please remember, it has to work for IPv6 and for IPv4 ... anyway, we can address these things during the review
This has been partially implemented with . Covering the code path for when an IP is does not exist on a node. I'll assign this back to you so you can split this issue or close it in favor of another issue. Thanks! /unassign /assign
what is the different codepath for ExternalIPs that already exist on the host? ~My reading is that the only difference is that it opens a local socket on that port, can you expand on this?~ Found it
so, it seems that this work in some environments because they are not using the bridge plugin why is not allowed that pods can reach ExternalIPs?
Wow, this is ancient and needs some fixing. This is not about preventing pod traffic from reaching those IPs. I had to really stare at it, and I am not 100% in my interpretation. First, we recognize such traffic from "off machine" (!physdev-is-in && !src=LOCAL, which we should probably switch to localDetector) and steer it to the Service. Second, we recognize ANY such traffic that is destined for a LOCAL address and shortcut that to the Service, Implicitly, traffic destined for this IP where the IP is not local goes to the system's default routing logic.

The reliance on the bridge plugin is awful and archaic and I will buy a beverage of your choice for whomever gets rid of that. I think localDetector can cover this, but not 100% sure
but the ExternalIP belongs to a service and we(kube-proxy) "knows" where to send it (to the service) , that is the part I'm not sure to understand, because whatever the user is doing with that ExternalIP it has to send it back to the cluster to reach the service. However, there is always context that is important and maybe I'm missing something. Assuming we want to handle ALL the ExternalIP traffic with kube-proxy, the logic will be much simpler: traffic to an externalIP will be ALWAYS sent to the Service (DNAT) if traffic is NOT LOCAL, we also SNAT
/remove-good-first-issue /remove-help-wanted


Question: "//staging/src/k8s.io/apiserver/pkg/apis/example/v1:go_default_library", "//staging/src/k8s.io/apiserver/pkg/endpoints/request:go_default_library", "//staging/src/k8s.io/apiserver/pkg/registry/rest:go_default_library", <ins> "//staging/src/k8s.io/client-go/kubernetes/scheme:go_default_library", </ins> "//vendor/github.com/evanphx/json-patch:go_default_library", "//vendor/github.com/google/gofuzz:go_default_library", "//vendor/k8s.io/utils/trace:go_default_library",

What happened: POST or PUT to a custom resource with a JSON body missing "kind" results in a 500 error: A missing "apiVersion" also results in a 500 error: Then with both "kind" and "apiVersion" defined we get the normal 4XX return codes. What you expected to happen: All bad inputs for create and update requests should be 4XX return codes. E.g. Creating a with body of results in the 422 error: How to reproduce it: Create a CRD or use a cluster that already has a one prepared (GKE 1.11.0+ clusters have the CRD installed which you could test against). HTTP to that custom resource with a JSON body of . E.g. Environment: Kubernetes version (use ): 1.11.2 Cloud provider or hardware configuration: GKE OS (e.g. from /etc/os-release): Kernel (e.g. ): /sig api-machinery <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
This is fixed on master, but I can reproduce it on 1.11. Posting a custom resource with a JSON body of : on master: on 1.11:
we are transforming all non-status error into 500 status error. in this case, it's the decoder returning a which is non-status. a quick fix by . (i like this number)
instead of fixed, it's rather been shadowed by i suppose. now when that empty unstructured obj going thru the decoder, the gvk is explicitly coerced, so this case is not reproducable on master for now.
/cc
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle rotten
I get the same response when trying to update the custom resource's 'status' sub-resource. On k8s v1.11.4
/assign
submitting the specified object on master returns a 400 badrequest, as desired /close
Closing this issue. <detailsIn response to : submitting the specified object on master returns a 400 badrequest, as desired Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "testing" "time" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" controllervolumetesting "k8s.io/kubernetes/pkg/controller/volume/attachdetach/testing" volumetesting "k8s.io/kubernetes/pkg/volume/testing"

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Background: The attach/detach controller create objects for CSI Volumes to indicate that a volume needs to be attached to a node. It deletes the object to indicate that the volume should be detached. Problem: If the Attach/Detach controller crashes, and in the meantime the pod using that volume is deleted, when the AD controller restarts, it won't know that the volume was previously attached and should be cleaned up. What you expected to happen: Proposal: We should have the AD controller periodically list all objects and for any that are not accounted for, update its internal actual/desired state caches so that the volume is gracefully detached. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
could I take it?
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/lifecycle frozen /reopen
Reopened this issue. <detailsIn response to : /lifecycle frozen /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
should address this