Question: {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"admin", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"{{kube_user}}", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kubelet", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} <ins> {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kube_proxy", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} </ins> {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"kubecfg", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"user":"client", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}} {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"group":"system:serviceaccounts", "namespace": "*", "resource": "*", "apiGroup": "*", "nonResourcePath": "*"}}  No newline at end of file

This is happening consistently across all GKE 1.4-1.6 master upgrade jobs. It is not occurring for full-cluster upgrades. Example log: https://k8s-
Config map is failing to mount for some reason.
Could you take a look. It seems like the dns pod wants a config map, but for the life of me, I can't find where this config map is created.
optional: true for the configmap means that if the configmap does not exist, then the configmap directory should be empty
the optional configmap feature was in 1.6
Ah. The kubelet here is 1.4 and doesn't understand optional configmaps. This causes the pod to error and never become ready.
cc
thanks, just saw this. glad to know the add-on manager support for ensureexists made it in 1.6
Glad to see addon-manager could help out instead of causing problems :)
merged, is this resolved now?
Seems like it is still failing in the latest run for the same reason: https://k8s- Checked the log and found the upgrade is using the beta release binary: So I assume the test will keep failing until that PR got cherry picked into 1.6 branch and released?
at least that, possibly until a new tag is cut
This issue appears to be .
we are now seeing similar-looking errors in the GKE cluster upgrade suites. Can you please take a look ASAP to help triage? Note that this time it is occurring in the full-cluster upgrades, not the master-only upgrades. Example: https://k8s-
Sure, looking into it.
FYI is looking at another issue related to GKE upgrades - not sure if they're related or not (they're reporting different errors)
I'm looking into an issue of the master not starting. This looks separate.
Don't have a theory yet, but saw couple strange errors: Which likely indicates Apiserver on master was not in a proper state. And seems like this happened before and after upgrade --- both old version and new version kubedns crashed with this error.
Found kube-proxy hit similar failure (seems like a permission error):
related to your ABAC work?
I just found this in a healthy master when trying to reproduce my problem. API server is definitely RBAC denying kube_proxy on cluster wide endpoint lists.

Was this an upgrade or a new install?
Almost certainly related to what I did. I checked a 1.5-1.6 upgrade, but didn't check 1.4-1.6
It was an upgrade.


Question: 203.0.113.1	podFoo.domainFoo	podFoo `, }, <ins> { "203.0.113.1", "podFoo", "domainFoo", []v1.HostAlias{ {IP: "123.45.67.89", Hostnames: []string{"foo", "bar", "baz"}}, }, `# Kubernetes-managed hosts file. 127.0.0.1	localhost ::1	localhost ip6-localhost ip6-loopback fe00::0	ip6-localnet fe00::0	ip6-mcastprefix fe00::1	ip6-allnodes fe00::2	ip6-allrouters 203.0.113.1	podFoo.domainFoo	podFoo 123.45.67.89	foo 123.45.67.89	bar 123.45.67.89	baz `, }, { "203.0.113.1", "podFoo", "domainFoo", []v1.HostAlias{ {IP: "123.45.67.89", Hostnames: []string{"foo", "bar", "baz"}}, {IP: "456.78.90.123", Hostnames: []string{"park", "doo", "boo"}}, }, `# Kubernetes-managed hosts file. 127.0.0.1	localhost ::1	localhost ip6-localhost ip6-loopback fe00::0	ip6-localnet fe00::0	ip6-mcastprefix fe00::1	ip6-allnodes fe00::2	ip6-allrouters 203.0.113.1	podFoo.domainFoo	podFoo 123.45.67.89	foo 123.45.67.89	bar 123.45.67.89	baz 456.78.90.123	park 456.78.90.123	doo 456.78.90.123	boo `, }, </ins> } for _, testCase := range testCases { <del> actualContent := string(hostsFileContent(testCase.hostIP, testCase.hostName, testCase.hostDomainName)) </del> <ins> actualContent := string(hostsFileContent(testCase.hostIP, testCase.hostName, testCase.hostDomainName, testCase.hostAliases)) </ins> assert.Equal(t, testCase.expectedContent, actualContent, "hosts file content not expected") } }

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? Ingress, hosts and /etc/hosts Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request There have already been a few issues regarding modifying : , But both have a different use-case. First some information before I explain my use-case. I have a pod which run a web app (nginx container + php-fpm container) and it available over multiple sub domains. I want request to the exposed sub-/domains (ex. ) to go directly to the local webserver instant of going the long way (Cloudflare -Ingress Controller -Node -Container). There is 2 reason behind this: is faster/reduce overhead. of our app code call the exposed domains, and need to hit the same server, as the request oriented from (the code write to some tmp file, and other PHP code need to read that tmp file, when it get a request. Bad design: properly). cc Kubernetes version (use ): Environment: DigitalOcean, CoreOS.
I'm not sure I understand the use case, especially the 2nd. This is the case of a pod talking to itself via the public hostname? Why not just use localhost?
Public hostname == ingress . Yes, without going through the load balancer, some of the PHP also require "sticky" sessions and then it is just easier if the exposed domains just point to localhost. That require extra logic in the app code and when the app code expose multiple sub-/domains, then it is just easier adding the domain to /etc/hosts.
Do you know all the hostnames at pod startup time, or can they possibly be later? I guess I am not against this idea in principle, I just have a nagging suspicion it will blow in all the corner cases. I guess we could look at a community implementation, if someone did one. I don't think that we have time on our side right now to drive it.
I currently have a script like this: , which "extract" the hostnames from the nginx config at container startup (and "patch" the location of the file). It also configure to forward some request internally (we don't expose all domains externally).

-- I'm not sure about your particular use case as the host name extraction logic seems to be quite complex (parsing of the nginx config). The two things I do see that are useful: Ability for a ConfigMap to be written out as a single file into the container instead of injecting it as a directory of files. This would solve a lot of problems where a piece of the container configuration needs to be trivially changed. I'm fairly certain this is already filed, I can't seem to find the issue. Inheritance of host entries from the outer node.
I think the use-case beyond service is if the hosts need to be loopback or needs to be configured per pod. I tried to understand how is controlled by kubelet, and it seems like this feature would not be hard to implement. I'm willing to spend some time on this. in terms of implementation: from my reading, host mount is done by inside , and the content of the hosts file is controlled by the methods that call them all have access to the pod object. it seems like we can add a field to PodSpec (can be done as an Pod-level alpha annotation to start): or if we want something more explicit: that can be read into and , which then mounts as to the Pod.
Let me know if ^ makes sense. If so, I can make a formal proposal in or just a PR, whichever is appropriate.
I don't think this warrants a proposal or feature. I would argue for the more expliciti form of the API, but hostnames has to be plural, and IP has to be the mergeKey. or - any objections? Consider from a CRI point of view?
For me, this solves my problems.
so the struct would be something like where the keys are IP addresses, and values are hostnames. I presume this field would live under PodSpec. is the key we want to surface? ? the content would be: which after some merging would look like: (the go struct represented in yaml) and the output could either look like: or I think the latter is better since it's one line per entry, and seems to follow existing convention in .
Your proposal can definitely work for my use-case, although I would love if I could configure it from a init container, but this could work.

Keep up the good work :)
I wouldn't try to massage it into a map, just: , Kristian Klausen
So you'd be passing in into . That makes sense. This might be a stretch, but what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP.
This is currently handled completely by kubelet, so it wouldn't be an issue for CRI. kubelet doesn't mount if the pod is using the host network, so this would either need to be changed, or validation should reject the pod if this is not allowed. If you need to dynamically change the file after the container starts, I think suggestion of trying to make ConfigMap work would be better.
I think validation to reject the option if the pod is using the host network makes sense. is still a static file though. And if you inject a file that replaces , what are you going to do with existing dynamic hostnames that are by kubelet?
I don't have any concern from CRI perspective, and the proposal look fine to me from the high level. what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP. But can I request not stretching the feature to support dynamic changes? Pod can be deleted, and re-created, thus the pod's IP might be changed. Currently we only allow container image being updated in PodSpec. Unless we allow updating the new field, let's call it etcHost for now, in PodSpec, it won't work. Even we allow that field being updated, I think that is overkill.
With regards to the dynamicness, I'm just trying to design the API in such a way that's flexible. The field in PodSpec won't be dynamic. I imagine something like this: (The idea is borrowed from ) validation ensures that is only present if type is . then IP is gathered at Pod creation time. In kubelet's :
Why do you need pod IP? Why not use 127.0.0.1 ?

, Ricky Pai <:
The use-case that i'm more interested in is support for nodeIP, but I'm using podIP as an example because it's easier to illustrate with current code. But yes I agree with you that pod IP might not make sense since you have access to localhost already. I can imagine nodeIP being used to talk to daemons that are shared across pods. is a proper fix for the use-case, but this could be one way to address it.
sorry for the scope-creep. I'm just afraid using IP as a string field makes it hard to add support for other IP sources should that ever be an option. but it's probably premature right now. for now, I'll just proceed with:
Given that we have a way to get node IP through downward API, why do we need that here? On Apr 13, 2017 6:09 PM, "Ricky Pai" <:
Ideally, if the applications running inside the containers have flexible ways to configure dependent hostnames like through environment variables, then downward API would be sufficient. We can probably extend the argument that the whole support for is moot if applications supported flexible injection of dependent hostnames. The problem is many applications don't provide this level of flexibility. Hostnames are often hard-coded inside config files and the client libraries. Outside of Kubernetes, I might use a script to manage and expect nothing else to change the file; while it is possible that I use an init container to do the same modification to , since the mounting of happens whenever a container is restarted, I risk the file being changed and my application not being able to connect to its dependent services. This creates much headache migrating applications from a more lenient environment to Kubernetes, which is what we've been facing. I think support is the right direction and is why I'm willing to commit time to make it happen.
We have another issue open for "node-local Services" which sounds like what you want. I am not a big fan of the expansion of host mappings like this. I'd rather go all the way and give nodes a static link-local address or else solve node-local services. , Ricky Pai <:
Thanks for all the feedback I'm well aware of node-local services, but it seems like progress on the proposal is stalled.

I believe node-local services would be the proper way to address the use case I described, especially with the "locality" concept you described in that can be expanded beyond the node. However, in lieu of node-local services, with references to the node IP might be a quick way to address the use case. Other tickets describing the need for node-level sidecars also proposed either using node-local services or : I also understand this complicates the interface a little bit more, but it would really benefit cluster operators. If you still think this is inappropriate, then I can proceed with this interface next week:
I don't have a strong opinion on this topic but it seems that it should be addressed as part of what is the status of the proposal?
I am not familiar with the node-local services and its progress, but if this can satisfy the use cases, I'd suggest that not adding new fields to the pod API for a quick workaround.
I feel the original proposal got digressed by my zealous talk of node IP support. Allow me to clarify: serves several use cases: external hostname with a specific IP. i.e. if I want to say mock out an external hostname like a hostname with localhost. i.e. if I want to override to point to we allow dynamic injection of nodeIPs, it can be an easy way for pods to talk to nodes in absence of node-local services. only the third use-case is relevant to node-local services. I think it's still useful to support adding entries to even with static IPs
I understand the third case, but I'd like to consider other options to solve that, first. I think the rest is uncontentious. , Ricky Pai <:
Thanks. I'll get the changes up this week. There'll likely be three changes: tests for existing hosts file content (there's no tests currently) change for on to use in Is there any possibility to cherry-pick those back in 1.6? Or would this have to wait for 1.7?
AFAIK, we don't cherrypick features.
has been created for the API change
just to clarify, are you in favor of adding HostMappings to the API?
I am neutral on it. It's come up often enough that i believe the problem is real. The API doesn't seem harmful to me, what do you think?

You're the guardian of CRI... , Yu-Ju Hong <:
Also the name is terrible - help? HostAliases? DNSAliases? , Tim Hockin <: solve that, first. I think the rest is uncontentious. , just to clarify, are you in favor of adding HostMappings to the API? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I'm up for . might accidentally imply that it's actually interfacing with kube-dns.
I am neutral but just wanted to make sure there are no obvious, alternative ways to achieve this. BTW, the proposed solution changes the k8s API, but not CRI :) On the other hand, if we support this on the k8s API level, we should discourage users from modifying this file directly. For example, reported a bug where users modify in the init containers, and later found that kubelet overwrites it.
We should definitely discourage modifying that, but we should perhaps respect it if a user provided a volume mount. Thoughts on name? , Yu-Ju Hong <:
I am terrible at naming... I agree that having DNS in the name can be misleading. sounds ok. might be more direct?
I'd go for brevity when possible. HostAliases ? On Apr 27, 2017 5:27 PM, "Yu-Ju Hong" <:
I'm updating the PR to use . thanks for the suggestion
Just posted the last PR to enable this feature:
could you add a node e2e test for this feature?
Do we need e2e? Every e2e is a liability, and slows down test runs - e2e is needed if we can't cover in unit or integration. , Yu-Ju Hong <:
Unit tests are great to ensure the individual functions work correctly, but so far it's not sufficient to prevent regression in our codebase from my experiences. Since this is a new addition/feature to the API, I'd prefer having a corresponding node e2e test. Integration test can be sufficient, but I've not had any pleasant experience debugging integration tests. Maybe that has changed already?
no, integrtaion tests are under-serviced tech-wise.

, Yu-Ju Hong <:
does a simple node e2e test sound good to you?
SGTM , Yu-Ju Hong <:
I saw that the HostAlias's comment: I'm wondering why it couldn't work to hostNetwork pods? And how does a hostNework pod initialize its /etc/hosts file?
Are we still missing a test for this? I'd hate to have to kick it out of the 1.7 release for lack of tests..
yes we are still missing a test. what's the deadline for 1.7 release? sorry i was on PTO and dropped the ball
hostNetwork pods uses the host's file, while only non-hostNetwork pods have managed by kubelet. because kubelet does not manage the file when using hostNetwork, kubelet can't write additional entries from hostAliases.
Just chatted on slack, for containers using the host network, docker actually makes a copy of "/etc/hosts" on the host for each of them. I think kubelet can potentially support HostAliases for hostnetwork pods by copying and mounting the file. Not sure if there are be other concerns though.
if you make the change to make kublet copy and manage for hostNetwork pods, I can make the change for HostAliases. one concern: currently, the logic includes , including from and . Should this be allowed? if not, there should probably be validation disallowing those field when hostNetwork is used. there currently isn't any unless I read the code wrong.
node e2e test:
THANKS! , Ricky Pai <:
That would be a great feature! Currently we have pods using hostNetwork and using init containers to restore some host mappings -- basically . With HostAliases feature it would be fantastic fit!
We are using and for hostNetwork pods.
This is something I don't understand in the code: It seems that it is still mounting "/etc/hosts" for hostnetwork pods if . Does that mean kubelet is still managing it?
One of our current assumptions is that kubelet will mount a shared /etc/hosts (which is not node's) for all containers within a pod in hostNetwork mode. Which seems to be the codepath in the code linked above. Is this holding true?
I checked the git history and that was just an artifact caused by switching from internal to v1 API objects.

The original code is: In the internal API, is a field in , so you can see why had to be non-nil. In the v1 API, is a field in the pod spec: . When switching all code to use the versioned (v1) objects, this got lost in translation. I believe the apiserver sets the default pod SecurityContext, so it should never be nil. I'll send a PR to remove the redundant check.
Thanks for looking into that. Currently even with hostNetwork pods, it will have mounts for . Checking within the pod: If the check is removed, what would happen on the shared mount of /etc/hosts ? My concern and question is about the guarantee here. Is there still a copy of which is shared and mounted for all containers within a pod?
Removing the check itself won't affect anything since securitycontext should always be non-nil. That mount for hostnetowrk pods was by docker (not by kubelet) so it should be one per container, and not shared by containers within a pod. I didn't try it myself, but it should be easy to verify.
Tried and verified that was shared between containers within a pod for k8s 1.6.4 .
Also verified that modifying for a hostNetwork pod did not change the file on the node.
Then docker did the right thing and use the same file for containers sharing the same network namespace :-)
That makes sense.
Will HostAlias support for HostNetwork pod happen in 1.7 ?
I'm trying to get in so it can make the 1.7 release deadline. help reviewing the PR would be great.
in a multi-container hostNetwork pod, what happens when you change the file, then restart one of the containers? in non-hostNetwork pod, it would revert the changes to
anyways, i'm also in favor of kubelet managing the file even for hostNetwork pods. that way, we can use features like hostAliases, and also then we don't depend on the container runtime to dictate the behavior of the hosts file. it's not clear whether running rktnetes with hostNetwork pod would yield the same result
Is this a dup of ? How are they different?
HostAlias is useful to provide local overrides for a particular Pod, but it seems like would want the alternative DNS name of a service accessible for all Pods.

re: documentation, i'm thinking either adding notes about HostAliases in or as a separate page under "Services, Load Balancing, and Networking"
Is there any issue created for supporting HostAlias in hostnetwork mode?
I can write up what I think will be needed to support HostAlias in hostNetwork pods
This is now in 1.7:
filed issues to follow up with HostNetwork support:
Thank you for this feature. Our use case is different. On an nginx/php-fpm set up where our local dev is via docker-compose yet we're deploying into kubernetes, we can use the exact same nginx config on both environments. We're currently working around this via HOSTALIASES (alias or or whatever the php container is into , which satisfies both deployment types), but this is hacky and doesn't work in Alpine.
the discrepancy between dev/prod would probably cause problems beyond just difference in networking setup. have you considered using to have parity between the two environments?
We have, and found it way too cumbersome for local development. It's way heavier in resources as well - obviously there's a lot more than just docker needed for minikube to run, and most of our developers use Mac, where all this is virtualised. In contrast, docker-compose is a lot easier, more concise, and we have yet to find any environment-y issues caused by difference between these two, except for this one issue we fortunately have a workaround for (HOSTALIASES). We use the exact same base container images for development as well as builds. From the point of view of the services themselves, there's no difference at all being run on one environment or the other - they talk to each other via HTTP or queues.
that's unfortunate. I guess minikube does have overhead since you have to run it inside a VM on your mac machines. But since all components are written in go and cross-compiled in darwin, I wonder if you could setup your own etcd (homebrew has this), then your own kubelet. Docker you can run Docker on Mac, then just have kubelet talk to the local docker socket. This sounds like it would work with lower resource overhead than minikube since you don't to run in a VM. Either way, I'm glad HostAlias is able to address your problem.

I just wish you didn't have to deal with this discrepancy in the first place.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close


Question: IgnoreNotFound *bool Now            *bool Timeout        *time.Duration <ins> Wait           *bool </ins> Output         *string }

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig CLI What happened: Running leads to nil pointer dereference: What you expected to happen: is expected to run delete with wait in that case How to reproduce it (as minimally and precisely as possible): (with any valid resource yaml) Anything else we need to know?: Discovered in when accidentally enabled wait for deletion from other commands than and saw failing integration tests in https://k8s- Environment: Kubernetes version (use ): latest master Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
This bug was introduced in : is set in that requires to be non-nil flag is being set in is normally being set in method which is not invoked in . a result, the invocation of resource finder leads to nil pointer dereference (see stack trace in the description).
/cc


Question: "k8s.io/legacy-cloud-providers/azure/clients/interfaceclient/mockinterfaceclient" "k8s.io/legacy-cloud-providers/azure/clients/publicipclient/mockpublicipclient" "k8s.io/legacy-cloud-providers/azure/clients/vmclient/mockvmclient" <ins> "k8s.io/legacy-cloud-providers/azure/clients/vmssclient/mockvmssclient" "k8s.io/legacy-cloud-providers/azure/clients/vmssvmclient/mockvmssvmclient" </ins> "k8s.io/legacy-cloud-providers/azure/retry" )

What happened: We saw several occurrences of kube-controller-manager failing to check a k8s node existence after we deleted that node's parent VMSS (and by extension, cloud instance). The leaked k8s nodes remained present in the cluster until manually deleted. Cloud provider retries caused a VMSS VMs API call surge, degenerating in hard throttling for the whole subscription. In those situations, kube-controller-manager would continuously log errors like the following: Looking at Azure cloud-provider implementation shows that all the VMSS API calls errors (including a 404/not found on the VMSS object) are bubbled up as generic errors, and never used as an indication of a missing instance. What you expected to happen: k8s nodes being removed as their VMSS went missing. How to reproduce it (as minimally and precisely as possible): Create and delete many VMSS providing instances registering to a cluster. Environment: Kubernetes version (use ): 1.19.0 Cloud provider or hardware configuration: Azure (self managed cluster, non AKS, ) OS (e.g: ): Linux, Ubuntu Focal 20.04.1 LTS Kernel (e.g. ): 5.4.0-1022-azure /sig cloud-provider /area provider/azure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: Type=oneshot RemainAfterExit=yes ExecStartPre=/usr/bin/mkdir -p /opt/cni <del> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-9d5e6e60e79491207834ae8439e80c943db65a69.tar.gz </del> <ins> ExecStartPre=/usr/bin/curl --fail --silent --show-error --location --create-dirs --output /opt/downloads/cni.tar.gz https://storage.googleapis.com/kubernetes-release/network-plugins/cni-07a8a28637e97b22eb8dfe710eeae1344f69d16e.tar.gz </ins> ExecStart=/usr/bin/tar xf /opt/downloads/cni.tar.gz -C /opt/cni/ - name: kubernetes-download-salt.service

I have a container that got the broadcast address of podCIDR subnet. My CNI conf is: And the container in question has which is the broadcast address for the subnet. So it can't figure out the MAC of cbr0 because all frames are getting sent to (or at least, that's what seems to indicate). Eg: tcpdump alonside a wget on vs a similar normal container
Looks like a bug in ipam. Will try to put together a fix.
I stress tested your fix overnight with the networking suite I'm writing (the one that found the bug), didn't hit the failure in 100 runs, we should get it in asap
Great. It would be better to pull from upstream once the fix is merged. Otherwise, we have to fork it like we did in 1.3.
Marking this p0 flake so it shows up in release queries
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.


Question: package scheduling import ( <ins> "fmt" </ins> "time" "github.com/onsi/ginkgo"

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: soak-gce-gci-1.13, others Which test(s) are failing: [sig-scheduling] SchedulerPredicates [Serial] validates MaxPods limit number of pods that are allowed to run [Slow] Since when has it been failing: Rougly forever. shows how this test was flaking back during the 1.11 cycle. There's been no sudden increase in frequency, but a test that flakes roughly 1.5 times per hour needs looking into, and probably tweaking. Testgrid link: https://k8s- Reason for failure: Most of the time, these flakes are timeouts. However, sometimes they have other causes; see Triage link: Anything else we need to know: This is part of a general effort to clean up our CI signal by eliminating longstanding, frequent flakes. <!-- DO NOT EDIT BELOW THIS LINE --/kind flake /priority important-longterm /sig scheduling attn:
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale /help-wanted
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
There are several different failures in this test, I was in particular looking at which usually boils down to something like this My hunch is that we are trying to schedule more than we should. This is how we compute how many pods to schedule which just takes max capacity and substracts whatever this returns My guess would be that the computation is not completely correct. My idea would be to add more logging into to be able to figure out more. Does that sound reasonable to you or would you take another approach? cc.

I went over all the pods that were on the node before the test started and then after and diff-ed all the pods that are not started by the test "grep -v maxp-" and did not found any pod that would start in the middle of test and was not present in the beginning :/ but there were 18 pods in the beginning, 330 is the full capacity so we should have started 312 to fully saturate, while we were trying to start 313. (maybe some pod restarted right when we were starting the test?) I am looking at this latest case but there's many like this one
Can we possibly be missing some pods here? pods with no node assigned that are not unschedulable?
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
This just got much worse in the last weeks, I tried looking into PRs along the time and hit into this could someone please verify it that could have made things worse? I'll also try to look tomorrow since I investigated this while ago :)
Anyone can help here? It's constantly red for several days, it's not even flaking anymore in our release dashboards /priority important-soon


Question: err := c.CoreV1().Namespaces().Delete(ns, nil) Expect(err).NotTo(HaveOccurred()) <del> err = framework.WaitForNamespacesDeleted(c, []string{ns}, 3*time.Minute) </del> <ins> err = framework.WaitForNamespacesDeleted(c, []string{ns}, framework.DefaultNamespaceDeletionTimeout) </ins> Expect(err).NotTo(HaveOccurred()) By("Verifying Persistent Disk detaches")

/assign The test flakes presubmit testing in PRs six times per week The site can focus on just this test: Which shows that 83 out of 85 errors match this pattern:
Previous attempt to address this flake:
CC
Took a quick look at a failed test: https://k8s- The indicates: . or and can you help investigate.
CC
The failure in the GKE multizone test suite is probably due to the GKE issue I fixed last week.
We'll dig into this and see what's up.
Expectations: This work should interrupt and take priority over other activity while it is assigned to you. Expectation for sig leads is to identify someone to fix the issue, them and yourself. Expectation for assignees are to investigate the issue and either a) resolve it or b) it to a more appropriate person and b) yourself.
I repeatedly ran this test on my e2e cluster since 7pm last night, and it has not failed yet. Seems like it fails more frequently in the PR env instead of the CI env, so may need to look into differences between the two. The test is failing during the deletion of the namespace. It seems like there are still some resources not getting deleted. I'm trying to dig deeper in the logs to see if I can find what resources are not getting cleaned up.
Here's one failed : From the API server initial namespace deletion request from the e2e test: I0427 23:23:29. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (4.744775ms) 200 [[ (linux/amd64) kubernetes/] 104.197.101.155:] controller triggers deletion on all the objects in the namespace: I0427 23:24:37. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-cx3bh/statefulsets: (91.176059ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] I0427 23:24:37.

5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh/resourcequotas: (7.608267ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] etc ... gets finally deleted by controller I0427 23:26:30. 5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-cx3bh/statefulsets: (87.184389ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serv iceaccount:kube-system:namespace-controller] [::1]:] I0427 23:26:30. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh/resourcequotas: (21.517822ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccoun t:kube-system:namespace-controller] [::1]:] etc ... deletion finished I0427 23:26:31. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (16.58598ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:namespace-controller] [::1]:] 2E test cleanup tries to delete it again: I0427 23:26:31. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-cx3bh: (9.104808ms) 404 [[ (linux/amd64) kubernetes/] 104.197.101.155:] <= This is the e2e test failure cleanup Total time to delete 3 mins Here's a successful : : deletion request I0428 00:45:14. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (11.208829ms) 200 [[ (linux/amd64) kubernetes/] 104.197.153.246:] starts object deletions I0428 00:45:46.

5 ] DELETE /apis/apps/v1beta1/namespaces/e2e-tests-pv-c4t8j/deployments: (33.104152ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:servi ceaccount:kube-system:namespace-controller] [::1]:] I0428 00:45:46. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j/podtemplates: (3.447937ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:k ube-system:namespace-controller] [::1]:] etc ... gets finally deleted by controller I0428 00:46:17. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j/podtemplates: (10.679029ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount: kube-system:namespace-controller] [::1]:] I0428 00:46:20. 5 ] DELETE /apis/extensions/v1beta1/namespaces/e2e-tests-pv-c4t8j/deployments: (6.130617ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system: serviceaccount:kube-system:namespace-controller] [::1]:] etc ... deletion is finished I0428 00:46:21. 5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (21.10359ms) 200 [[kube-controller-manager/v1.7.0 (linux/amd64) kubernetes//system:serviceaccount:kube-system:na mespace-controller] [::1]:] 2E test cleanup I0428 00:46:32.

5 ] DELETE /api/v1/namespaces/e2e-tests-pv-c4t8j: (1.436939ms) 404 [[ (linux/amd64) kubernetes/] 104.197.153.246:] Total time to delete ~1 minute Comparing the bad and good runs, these were some supicious areas: the bad run, the period between 1) and 2) was 1 minute, whereas in the good run it was 30 seconds the bad run, the period between 2) and 3) was ~2 minutes, whereas in the good run it was ~20 seconds. The slowness in the bad run seems systematic, I did not find any one object that was taking a particularly long time to delete, all of them seemed to be taking longer compared to the good run. I'm not sure where to dig down further to find the cause of the general slowness. I don't know if this is expected, and we should just increase the timeout? The test framework seems to have a default timeout of 5 minutes, whereas this volume test timeout is only 3 minutes.
Great analysis! Since no singular event stands out as taking too much time how about, in the name of expediency, just changing the timeout to 5m and see how the tests behave?
The delay of ns deleteion might related to this PR and . is still open.
Those two issues do not seem to impact the deletion of the namespace itself, so I don't think fixing those will solve this issue.
But I think delaying the deletion of a pod will affect the time to delete the namespace.


Question: t.Errorf("unexpected deletesnexpected: %vn  actual: %v", expected.List(), actual.List()) } } <ins> func TestStoreDaemonSetStatus(t *testing.T) { getError := fmt.Errorf("fake get error") updateError := fmt.Errorf("fake update error") tests := []struct { name                 string updateErrorNum       int getErrorNum          int expectedUpdateCalled int expectedGetCalled    int expectedError        error }{ { name:                 "succeed immediately", updateErrorNum:       0, getErrorNum:          0, expectedUpdateCalled: 1, expectedGetCalled:    0, expectedError:        nil, }, { name:                 "succeed after one update failure", updateErrorNum:       1, getErrorNum:          0, expectedUpdateCalled: 2, expectedGetCalled:    1, expectedError:        nil, }, { name:                 "fail after two update failures", updateErrorNum:       2, getErrorNum:          0, expectedUpdateCalled: 2, expectedGetCalled:    1, expectedError:        updateError, }, { name:                 "fail after one update failure and one get failure", updateErrorNum:       1, getErrorNum:          1, expectedUpdateCalled: 1, expectedGetCalled:    1, expectedError:        getError, }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { ds := newDaemonSet("foo") fakeClient := &fake.Clientset{} getCalled := 0 fakeClient.AddReactor("get", "daemonsets", func(action core.Action) (bool, runtime.Object, error) { getCalled += 1 if getCalled <= tt.getErrorNum { return true, nil, getError } return true, ds, nil }) updateCalled := 0 fakeClient.AddReactor("update", "daemonsets", func(action core.Action) (bool, runtime.Object, error) { updateCalled += 1 if updateCalled <= tt.updateErrorNum { return true, nil, updateError } return true, ds, nil }) if err := storeDaemonSetStatus(fakeClient.AppsV1().DaemonSets("default"), ds, 2, 2, 2, 2, 2, 2, 2, true); err != tt.expectedError { t.Errorf("storeDaemonSetStatus() got %v, expected %v", err, tt.expectedError) } if getCalled != tt.expectedGetCalled { t.Errorf("Get() was called %v times, expected %v times", getCalled, tt.expectedGetCalled) } if updateCalled != tt.expectedUpdateCalled { t.Errorf("UpdateStatus() was called %v times, expected %v times", updateCalled, tt.expectedUpdateCalled) } }) } } </ins>

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Almost every update to a DaemonSet can lead to 2~3 error logs in kube-controller-manager even when the DaemonSet manages very few Pods, for example, there were 3 error logs when updating a daemonset in a small cluster of 2 Nodes. Other controllers like deployment don't have this issue when they manage much more Pods. After looking at DaemonSet controller code, it seems the retry logic in "storeDaemonSetStatus" was not implemented correctly. It never really retried and always made an useless GET call. DaemonSet controller should produce less error logs in normal case. Create or update a DaemonSet to trigger Pod recreation. Kubernetes version (use ): v1.20.4 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps /assign created to fix the retry logic.
Same problem with me. </brCould you explain why other controllers(like deployment controller) is not easy to get conflict when updating status? I think there are many in every controller.
the number of workers doesn't matter here as the workqueue guarantees that they will not process the same resource at the same time. The conflict comes from multiple successive processing of a resource because of the update delay of the resource's cache in . For example, when a DaemonSet manages 2 Pods and they are getting ready. first Pod is ready and triggers updating Daemonset, whose resourceVersion is in Lister. updates DaemonSet's ready count to 1 with resourceVersion set to and it succeeds. The resourceVersion in server becomes and the Informer/Lister hasn't received the update.

second Pod is ready and triggers updating Daemonset, whose resourceVersion is still in Lister. updates DaemonSet's ready count to 2 with resourceVersion set to and it fails with the "the object has been modified" error. To avoid this error, retry is the most straightforward way but it is implemented differently in controllers. In deployment controller, it actually has the same conflict but it only logs the error with so normally you don't see it, and it requeues the resource and re-processes it upon conflict. DaemonSet controller and Replicaset controller retry directly when calling fails. However, the former made a mistake when checking the loop's boundary condition and it never retried. is fixing it.
Very clear explanation. Thanks.


Question: Update(id string, obj interface{}) Delete(id string) List() []interface{} <del> Contains() util.StringSet </del> <ins> ContainedIDs() util.StringSet </ins> Get(id string) (item interface{}, exists bool) // Replace will delete the contents of the store, using instead the

The client/cache interface has a method which is supposed to return a of the IDs in a store. This is a bit misleading and should probably be renamed to 'Contents' or 'ContainedIDs'.
Thoughts
I tend to agree. contains to me should always return a bool and take an input key. Sent from my iPhone
Yeah, I agree, both of the suggested names would be an improvement.


Question: } return selector, nil } <ins> func isSubset(subSet, superSet labels.Set) bool { if len(superSet) == 0 { return true } for k, v := range subSet { value, ok := superSet[k] if !ok { return false } if value != v { return false } } return true } </ins>

What happened: The function return when whitelist is empty. The function is implemented as follows: What you expected to happen: When the whitelist is empty, any key of labels can not in it. Should the function return when whitelist is empty?
/sig api-machinery
/assign /cc (we think you might have opinions on fixing this)
I only see one consumer. Rather than change the semantic, I would suggest moving it to the point of use and renaming it.
Yes, this function is wrong, I agree with David. P.S. we don't advertise interface stability for this library so this is OK. But even if we did, I think callers of this function are probably not expecting this behavior and should be examined.) P.P.S. The way it should work is via set difference, the labels should be a subset of the list. P.P.P.S. These are sets and not lists, and the names should be changed to reflect that. P.P.P.P.S. If you're making a change here anyway, please lose the word "whitelist".
Okay, I will submit a pr.


Question: key := buildVmssCacheKey(nodeResourceGroup, ss.makeVmssVMName(ssName, instanceID)) defer ss.vmssVMCache.Delete(key) <del> klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s)", nodeResourceGroup, nodeName, diskName) </del> <ins> klog.V(2).Infof("azureDisk - update(%s): vm(%s) - attach disk(%s, %s)", nodeResourceGroup, nodeName, diskName, diskURI) </ins> _, err = ss.VirtualMachineScaleSetVMsClient.Update(ctx, nodeResourceGroup, ssName, instanceID, newVM) if err != nil { detail := err.Error() if strings.Contains(detail, errLeaseFailed) || strings.Contains(detail, errDiskBlobNotFound) { // if lease cannot be acquired or disk not found, immediately detach the disk and return the original error <del> klog.Infof("azureDisk - err %s, try detach disk(%s)", detail, diskName) </del> <ins> klog.Infof("azureDisk - err %s, try detach disk(%s, %s)", detail, diskName, diskURI) </ins> ss.DetachDiskByName(diskName, diskURI, nodeName) } } else { <del> klog.V(2).Infof("azureDisk - attach disk(%s) succeeded", diskName) </del> <ins> klog.V(2).Infof("azureDisk - attach disk(%s, %s) succeeded", diskName, diskURI) </ins> } return err }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: add more logging in azure disk attach/detach What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: /sig azure /assign
The issue message here doesn't say why a change was desired. The PR "Original logging is not clear when dealing with managed disk, since diskName is empty". That's more informative.


Question: func TestPollMinions(t *testing.T) { table := []struct { <del> minions []api.Node </del> <ins> minions       []api.Node expectedCount int </ins> }{ { minions: []api.Node{ <del> {ObjectMeta: api.ObjectMeta{Name: "foo"}}, {ObjectMeta: api.ObjectMeta{Name: "bar"}}, </del> <ins> { ObjectMeta: api.ObjectMeta{Name: "foo"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{ {Kind: api.NodeReady, Status: api.ConditionFull}, }, }, }, { ObjectMeta: api.ObjectMeta{Name: "bar"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{ {Kind: api.NodeReachable, Status: api.ConditionFull}, }, }, }, { ObjectMeta: api.ObjectMeta{Name: "baz"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{ {Kind: api.NodeReady, Status: api.ConditionFull}, {Kind: api.NodeReachable, Status: api.ConditionFull}, }, }, }, { ObjectMeta: api.ObjectMeta{Name: "baz"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{ {Kind: api.NodeReady, Status: api.ConditionFull}, {Kind: api.NodeReady, Status: api.ConditionFull}, }, }, }, </ins> }, <ins> expectedCount: 4, }, { minions: []api.Node{ { ObjectMeta: api.ObjectMeta{Name: "foo"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{ {Kind: api.NodeReady, Status: api.ConditionFull}, }, }, }, { ObjectMeta: api.ObjectMeta{Name: "bar"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{ {Kind: api.NodeReady, Status: api.ConditionNone}, }, }, }, }, expectedCount: 1, }, { minions: []api.Node{ { ObjectMeta: api.ObjectMeta{Name: "foo"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{ {Kind: api.NodeReady, Status: api.ConditionFull}, {Kind: api.NodeReachable, Status: api.ConditionNone}}, }, }, }, expectedCount: 1, }, { minions: []api.Node{ { ObjectMeta: api.ObjectMeta{Name: "foo"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{ {Kind: api.NodeReachable, Status: api.ConditionFull}, {Kind: "invalidValue", Status: api.ConditionNone}}, }, }, }, expectedCount: 1, }, { minions: []api.Node{ { ObjectMeta: api.ObjectMeta{Name: "foo"}, Status: api.NodeStatus{ Conditions: []api.NodeCondition{}, }, }, }, expectedCount: 1, </ins> }, }

With latest master it can happen that pods get scheduled on minions that are not up and get stuck in state. Also the command now also lists the down minions.
Master now returns all minions, with minion status set. Scheduler hasn't been updated yet. I'll put a fix for this. kubectl should return only running minions, and take a '-a' option to list all minions now. This also needs to be fixed.
, Deyuan Deng wrote:
Yeah, the main concern is that minion status is not as clear as Pod, e.g. running minion may not be eligible to schedule a pod. I'm still trying to figure out the best way to convey the message to client.
Re-open as now list all minions, including down ones. We should probabaly first filter out unhealthy ones in kubectl, before we can show any status.
What do you mean by "filter out"? We should be caching minion status in the apiserver. As proposed, we should show all minions that exist, including unresponsive ones.
I mean do not show down minions for now for consistency (users do not expect down minion to be listed). I agree with what proposed, but I'm trying to figure out how / what we convey the status message to user. NodePhase + NodeConditions are essential for k8s components, but are we going to expose all such data to user?
Why wouldn't users need to see down minions? I would think as an admin i would want to see phase + conditions
nvm My original concern was that phase and conditions are meant to be used by k8s, so we may need some translation to make it easier to understand for admin. Also, they are not fully populated, so I thought we may delay showing status for now. I certainly have no problem showing down minions and adding status. I've created a PR


Question: exit 1;; esac <ins> NODE_SUBNETWORK=${IP_ALIAS_SUBNETWORK} echo "Using IP Aliases subnet ${NODE_SUBNETWORK}" </ins> # Look for the alias subnet, it must exist and have a secondary # range configured. local subnet=$(gcloud beta compute networks subnets describe

The "Services should be able to create an internal type load balancer [Slow]" e2e test is failing on our 2k-node gce clusters - https://k8s- with the following error: We're using custom subnets for our large cluster tests, and seems like the glbc controller is not able to handle them. cc
Seems similar to Assigning to you as you seem to have the context already :) Also, is this expected to be fixed in 1.8 timeframe?
doesn't set the necessary variables for the GCE cloud provider. Will throw you a PR.
Cool - thanks a lot!
Ref
I see the last test on 09/09 failed, but I can't see the build log due to exceeding memory. Any other way I can see the log, /reopen
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/network sig/scalability: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
nevermind - went directly to raw log.
Friendly ping. This e2e is still failing (https://k8s-) and it's part of release-blocking suite.
/close


Question: // getPhase returns the phase of a pod given its container info. func getPhase(spec *api.PodSpec, info api.PodInfo) api.PodPhase { <del> if info == nil { return api.PodPending } </del> running := 0 waiting := 0 stopped := 0

I have a cluster with ~130 pods. Running back-to-back 'get pods' shows me wildly different understandings of their state, even though I can tell from docker that they have not crashed or anything. thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 127 Running 1 STATUS 6 Unknown thockin kubernetes master /(NF)}' uniq -c Running: ./cluster/../cluster/gce/../../output/dockerized/bin/linux/amd64/kubectl get pods -l app=hostnames 82 Running 1 STATUS 51 Unknown Nothing changed in between these calls.
cc I blame the pod cache.
has recently changed pod cache? FWIW, I also observed this behavior: 19 pods Running in one call and all Unknown in the next. With 0.10.0.
Yes, I think PR on pod cache should fix the problem reported here. Also it should fix the issue reported by about adds-on services. I have a pending PR to have kubelet deciding on PodStatus too.
Could this have caused ?
I don't see how. I'll comment on that issue.
I don't think this could be the cause of that one.
I don't think PR on pod cache will fix this. That one clears the node status. This issue is with pod going from Running to Unknown even when nodes keep running just fine.
Totally consistent observation based on my personal experience. I just assumed we were shooting for some kind of Hesinberg Uncertainty Principle. S , Tim Hockin wrote:
Agree with clears node status only. One possibility is that we hit transient network problem in between the calls that fails health check, so controller set node condition to not reachable, hence unknown pods. I'll make this more robust anyway. Other than that, it's hard to think of a reason that node status results in the flakiness.
In the absence of node information, the pod status defaults to Unknown. Maybe there's another source of this specific problem, but the pod cache is a source of continuous problems, partly because it's not just a cache, but is implementing or duplicating a significant chunk of system logic in a half-baked manner. My goal is to eliminate it -- see .

Short of that, we need to at least implement , eliminate nodes from the pod cache, and implement a real node cache.
I can look into this. Repro info from Tim, who said he was running on a one-node cluster but has seen the problem with multi-node clusters also: run-container hostnames --image=kubernetes/servehostname --replicas=1 --port=9376 --labels=app=hostnames expose hostnames --port=80 --container-port=9376 --selector="app=hostnames" resize --replicas=150 rc hostnames but I had to hack the node a bit to make it get past 100 on the node: sysctl -w fs.inotify.maxuserinstances=8192 echo "DOCKERNOFILE=" /etc/default/docker; /etc/init.d/docker restart (fails sometimes, dunno why)
I can verify that the situation is rather bad. Lots of fluctuation in the numbers of Pending, Running, and Unknown. serve_hostname containers on the node are exiting; I'm not sure why, or if the pod itself is terminating or just the container. Lots of the following in the log (not sure what this means yet): Sat, 07 Feb 2015 00:06:22 +0000 hostnames-ecd2r Pod failedScheduling {scheduler } Binding rejected: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]
The pause container is also exiting, so I'm guessing that means the whole pod is exiting, not just serve_hostname...
Ugh, there is an etcd size limit you might be hitting too, I forgot. If you are seeing the fluctuations, maybe you got far enough, though. On Feb 6, 2015 5:13 PM, "David Oppenheimer" wrote:
Just a few more observations: This is reproducible (and a little less chaotic) with one node and 100 pods (but not with 50 pods, so it's definitely load-dependent). You can get to 100 Running pods (0 Pending or Unknown), and then a minute later a dozen or so will become Unknown, and then later go back to Running, etc. As mentioned earlier, the containers continue running on the node all this time (sudo docker ps wc -l), so it is definitely some kind of reporting problem.

As an aside, it looks like there is some kind of race condition with IP address allocation, as during startup (but not at steady state) a number of pods exit with the following in their log (sudo docker logs xxx): listen tcp :9376: bind: address already in use They do restart OK (I'm not sure if this is being restarted locally on the node or making a round-trip back to the scheduler).
Almost certainly the restart is just kubelet retrying after some time. On Feb 8, 2015 2:57 PM, "David Oppenheimer" wrote:
I found that if I comment out the line with the arrow next to it (see below) in , the Pods stop oscillating between Running and Unknown. Commenting out this line changes the logic from "if get an error from Kubelet, change pod status to Unknown" to "if get an error from Kubelet, keep pod status unchanged." I saw in the master logs intermittent errors of this form, which was causing the pods to flip back to Unknown: E0209 04:19:17. 4996 ] Error getting info for pod default/hostnames-v5drx: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-v5drx This seems to correspond to lines like this in the kubelet logs: I0209 06:38:11. 4704 ] GET /stats/default/hostnames-gfw5j/-b026-11e4-a070-/hostnames?num_stats=33: (218.857218ms) 500 goroutine 730 [running]: [ ... stack trace, but they're not always the same; I'll dig into this some more, the default stack trace depth wasn't going back past the various functions that just serve up the error ... ] However, pods still flip between Running and Pending, even though no containers are exiting on the minion, so there's another problem here too.
Thanks for investigating this! Pods really shouldn't flip between Running and Pending, as I believe that the only way that they go Pending is if Pod.Status.Host is empty, and this really shouldn't happen post scheduling, unless we are somehow dropping information. Maybe we need to make sure that we don't wipe out status on the error case?

--brendan , David Oppenheimer
I also investigated the same issue last Friday. I observed exact the same as what described above, except the related kubelet log should be: Meanwhile, I observed master is trying to query pod / and failed all the time: Talked to offline, he is going to look into why running pods go back to Pending; meanwhile, I am going to figure out Unknown cases here.
That would be great. It is very hard for me to make sense of any of the scalability experiments that I am doing while the pod status is in hysteresis.
Just to summarize: (1) the Running<-Unknown flapping is caused by the line of code I mentioned earlier; (2) the reason that line of code is being hit on the master is due to two errors on the kubelet (the one I identified earlier was wrong; Dawn found these): I0209 06:42:09. 4704 ] GET /api/v1beta1/podInfo?podID=hostnames-v5drx&podNamespace=default: (8.82543048s) 500 E0209 04:34:13. 4996 ] Error getting info for pod default/hostnames-2y8ny: kubelet "kubernetes-minion-1.c.striped-torus-760.internal" server responded with HTTP error code 500 for pod default/hostnames-2y8ny (3) we don't yet know the cause of the Running<-Pending flapping; I'm going to look into that now
(and Dawn is looking into why we're hitting those two errors on Kubelet)
Are we serving requests concurrently in Kubelet? (Is it possible not to when using go's servemux?) I don't see any locking around the pod map accesses.
This was marked for the fixit but Dawn and I are working on it. Assigned to myself.
I am also investing time writing code to try and report what is going on with unstable cluster status. It would be good to know what you and Dawn are doing in case I am wasting my time. Thanks. , David Oppenheimer
This particular problem is easily reproduced manually using the instructions Tim listed, so I'm not sure it's necessary to write anything sophisticated to drive the cluster into this state.

Anyway, maybe give me and Dawn a couple of days and if we can't figure this out by then we can call in the cavalry.
Am I wrong, or are these instructions also within bounds of an e2e test? Just because our existing e2es only do one or two things doesn't mean we can't do more within O(minutes), assuming you're dealing with the same image.
The "flips back to pending" problem appears to be because updatePodStatus() writes to the cache but not back to the Pod's status. computePodStatus() initializes newStatus based on pod.Status, not based on the cached value of status. This second behavior seems wrong in light of the first behavior. However, I believe this only causes a problem in combination with my fix described earlier, which as a result doesn't actually do what I described (it doesn't "keep the status as-is" but rather copies it out of the pod's status; to "keep the status as-is" it should be copying it out of the cache). I can see if this fixes things. (We also should fix the root cause stuff on the node side, which Dawn is working on.)
Assign the issue to me ... I didn't see the flips between Pending and Running anymore. Don't know why. I consistently see flips between Unknown and Running on those heavily loaded nodes. The root issue is once a while docker inspect API call returns us EAGAIN. Please note that on my test nodes, I already increase NOFILE limit for docker daemon process. The proposed solution is that kubelet deals with all errors internally except EAGAIN one. In this case, computePodStatus logic in pod_cache does nothing when p.containerInfo.GetPodStatus returns an error. Working on a PR to fix this.
Dawn, my suggestion is that computePodStatus() should return the current cached value whenever it gets an error from Kubelet. I can implement this if you want.
, why shouldn't updatePodStatus simply write through to the pod's status? I'd think either we always read the cached value first, or we always write through to the pod and read from there. Only reading the cached value upon an error seems counter-intuitive to me...
why do you computePodStatus returns a cached value here?

The only place calls computePodStatus is updatePodStatus, which triggered by two operations: 1) periodically update all containers 2) the client asks for PodStatus, and there is no cached one, instead calling computePodStatus to retrieve one. I think a cleaner way is 1) Kubelet handles all the errors except this transient one. Of course, we need to make sure docker is auto-restart and there is no too many open files error on node side too. But they are separate issues and should be handled by node side through script or configure etc. 2) Now back to master side, if node status is ok, kubelet should return something; if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round. I have one working PR to avoid the flips between Running and Unknown, but with increasing number of Pending from node side caused by docker failure. Still investigating it...
I didn't understand "if there is an error from kubelet, it should be transient; and if there is a cached one, simply return cached one upon request; if no cached one, it should be generated by computePodStatus in next round." If you're talking about the caching in the podCache on the master, then what you wrote is exactly what I was suggesting. (If you're talking about some kind of caching on the Kubelet, then that's different, because I am not aware of cache for status on the Kubelet.) I think always writing back to the pod's status might be fine but I don't know why the podCache was designed the way it was, so I didn't want to change that without asking Daniel first (I think he's the one who created it). I wasn't suggesting to only read the cached value upon an error; I meant always read the cached value under all circumstances.
and here is my understanding. Correct me if I am wrong. UpdateAllContainers periodically updates all containers, but it gets the initial pods from etcd through this line . each from , UpdateAllContainers calls passes to . computes and returns a new status. Note that the status is computed based on , i.e., the information in etcd. updates the pod cache with the new status.

Note that the new status is not written to etcd. If you repeat the same steps again, you will start with the stale pod status in the etcd because does not check the cached value.
I have tested since last night. The flaps between Running, Pending and Unknown are gone for about 10 hour running 300 pods on 4 nodes except two scenario: 1) etcd issue: cat kube- | grep "status to unknown" I0212 20:18:30. ] node doesn't exist: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0] <nil, setting pod hostnames-yuy8d status to unknown ... This happens a couple of times last 10 hours. We need to figure out why. 2) NodeStatus becomes UNREADY. It happens once last 10 hours, and pods status on that node become Unknown which is expected. I didn't figure out why NodeStatus was UNREADY then, but I heard there is a network earlier this morning.
up to you whether to reopen this issue or file new ones for new issues.
I've also seen the "All the given peers are not reachable" message (see my second entry in this issue). I think that was what was referring to when he said "there is an etcd size limit you might be hitting too" but I may be mis-interpreting.
That error is the etcd client's catch-all for "something went wrong" - it's actually really bad that it obscures errors. , David Oppenheimer
, Tim Hockin wrote:
and Yes, that error message is cache-all error message, which need to investigation. In my particular case, it shouldn't be etcd size limits at all since we have that cluster built and upgraded more than 10 hours before, and all 300 pods were scheduled back then. There is no new events generated either from kubelet or other components since all bound pods are indeed happily running on the nodes. No new writes to etcd except NodeStatus, which is very less frequent and not much data. The problem of etcd here occurs when master tries to read from etcd with an error.


Question: return printer.PrintObj(obj, o.Out) } <del> deleteOpts, err := o.DeleteFlags.ToOptions(dynamicClient, o.IOStreams) if err != nil { return err </del> <ins> o.DeleteOptions = &cmddelete.DeleteOptions{ CascadingStrategy: metav1.DeletePropagationBackground, DynamicClient:     dynamicClient, GracePeriod:       -1, IgnoreNotFound:    true, IOStreams:         o.IOStreams, Quiet:             o.Quiet, Timeout:           time.Duration(0), WaitForDeletion:   false, </ins> } <del> deleteOpts.IgnoreNotFound = true deleteOpts.WaitForDeletion = false deleteOpts.GracePeriod = -1 deleteOpts.Quiet = o.Quiet o.DeleteOptions = deleteOpts </del> return nil }

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: "k8s.io/apimachinery/pkg/util/diff" "k8s.io/apiserver/pkg/apis/audit" // import to call webhook's init() function to register audit.Policy to schema _ "k8s.io/apiserver/plugin/pkg/audit/webhook"

Currently the [audit policy decoder][1] silently drops unknown fields. For backwards compatibility, we probably cannot fail on unknown fields, but it would be good to at least log a warning when they're encountered. [1]: /sig api-machinery
/triage accepted
/help
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm not familiar with apiserver, but this seems like a simple one. /assign
How can we find unknown fields in the yaml file? Currently I'm think of converting both the yaml file and the decoded Policy struct to two map[string]interface{}, and compare both map to find out which keys are in the yaml file but not in the decoded struct, is there any better solution?
Courtesy on slack:


Question: func getFlexDir(c clientset.Interface, node *v1.Node, vendor, driver string) string { volumePluginDir := defaultVolumePluginDir if framework.ProviderIs("gce") { <del> if node == nil && framework.MasterOSDistroIs("gci", "ubuntu") { v, err := getMasterVersion(c) if err != nil { framework.Failf("Error getting master version: %v", err) } if v.AtLeast(versionutil.MustParseGeneric(gciVolumePluginDirVersion)) { volumePluginDir = gciVolumePluginDir } else { volumePluginDir = gciVolumePluginDirLegacy } } else if node != nil && framework.NodeOSDistroIs("gci", "ubuntu") { if getNodeVersion(node).AtLeast(versionutil.MustParseGeneric(gciVolumePluginDirVersion)) { volumePluginDir = gciVolumePluginDir } else { volumePluginDir = gciVolumePluginDirLegacy } } </del> <ins> volumePluginDir = gciVolumePluginDir </ins> } flexDir := path.Join(volumePluginDir, fmt.Sprintf("/%s~%s/", vendor, driver)) return flexDir

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: I think these jobs do not have the flexvolume directory properly configured. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/assign


Question: import ( "fmt" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/uuid" "k8s.io/kubernetes/test/e2e/framework"

Using the list of tested stable endpoints referenced in , we can pick out the endpoints which are part of the ConfigMap kind: This is a stable core endpoint which has existing testing, potentially making for low-hanging fruit to upgrade to Conformance. /area test /area conformance /sig testing
/assign
Promoting will increase Conformance Coverage by 1 endpoint.
PR was closed because AdmissionWebHook is not yet GA. /close
You can't close an active issue/PR unless you authored it or you are a collaborator. <detailsIn response to : PR was closed because AdmissionWebHook is not yet GA. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This test uses a beta version of the API. According to the , this is prohibited. The URL of the API which is test is hitting is .
/unassign The original PR was superseded by 's


Question: # For each release create a tmp dir 'etcd_release_tmp_dir' and unpack the release tar there. for version in $(BUNDLED_ETCD_VERSIONS); do  etcd_release_tmp_dir=$(shell mktemp -d);  <del> docker run --interactive -v $${etcd_release_tmp_dir}:/etcdbin golang:$(GOLANG_VERSION) /bin/bash -c  </del> <ins> docker run --interactive -v $${etcd_release_tmp_dir}:/etcdbin golang:$(GOLANG_VERSION)$(DOCKER_VOL_OPTS) /bin/bash -c  </ins> "git clone https://github.com/coreos/etcd /go/src/github.com/coreos/etcd  && cd /go/src/github.com/coreos/etcd  && git checkout v$${version}  && GOARM=$(GOARM) GOARCH=$(ARCH) ./build  && cp -f bin/$(ARCH)/etcd* bin/etcd* /etcdbin; echo 'done'";  <del> cp $$etcd_release_tmp_dir/etcd $$etcd_release_tmp_dir/etcdctl $(TEMP_DIR)/;  cp $(TEMP_DIR)/etcd $(TEMP_DIR)/etcd-$$version;  cp $(TEMP_DIR)/etcdctl $(TEMP_DIR)/etcdctl-$$version;  </del> <ins> $(BIN_INSTALL) $$etcd_release_tmp_dir/etcd $$etcd_release_tmp_dir/etcdctl $(TEMP_DIR)/;  $(BIN_INSTALL) $(TEMP_DIR)/etcd $(TEMP_DIR)/etcd-$$version;  $(BIN_INSTALL) $(TEMP_DIR)/etcdctl $(TEMP_DIR)/etcdctl-$$version;  </ins> done # Add this ENV variable in order to workaround an unsupported arch blocker

Part of What happened: Attempt to run as a non-root user Get What you expected to happen: It runs How to reproduce it (as minimally and precisely as possible): /sig release /sig cluster-lifecycle /area security And as part of kubeadm v1.16 milestone: /milestone v1.16 /priority critical-urgent
/assign randomvariable /lifecycle active
/priority important-soon


Question: return pod, err } pod.CurrentState.Info = info <ins> pod.CurrentState.Status = makePodStatus(info) </ins> pod.Kind = "cluster#pod" return pod, err }

ci-cgroupv1-containerd-node-arm64-al2023-e2e-ec2-eks E2eNode Suite: [It] [sig-node] Summary API [NodeConformance] when querying /stats/summary should report resource usage through the stats api { failed [FAILED] Timed out after 180.001s. Expected <string: Summary to match fields: { .Node.SystemContainers: missing expected element runtime } In [It] at: @ 07/26/23 05:48:05.248 } arm test since begining cadvisor on arm? /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign it may be ec2 related, not arm related. We need to fix arm tests on gce to validate (failing currently).
Should be fixed, but I'm watching and will re-open if not.
Test is passing now, good to leave closed.


Question: } func TestKubeletDefault(t *testing.T) { <ins> var resolverConfig string if isSystemdResolvedActive, _ := isServiceActive("systemd-resolved"); isSystemdResolvedActive { // If systemd-resolved is active, we need to set the default resolver config resolverConfig = kubeletSystemdResolverConfig } </ins> tests := []struct { name       string clusterCfg kubeadmapi.ClusterConfiguration

What happened: TestKubeletDefault unit test fails when run locally on system that has systemd-resolved active What you expected to happen: Unit test succeeds How to reproduce it (as minimally and precisely as possible): Use ubuntu with systemd-resolved active. Run Output: Anything else we need to know?: Environment: Kubernetes version (use ): Master branch (v1.19.0-alpha) Cloud provider or hardware configuration: n/a OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node /priority important-soon I created this issue to track what happened. I'll have a PR to fix soon. /assign
/area kubeadm /sig cluster-lifecycle
/remove-sig node
thanks for reporting. /remove-kind bug /kind failing-test /assign


Question: fsckErrorsUncorrected = 4 // place for subpath mounts <ins> // TODO: pass in directory using kubelet_getters instead </ins> containerSubPathDirectoryName = "volume-subpaths" // syscall.Openat flags used to traverse directories not following symlinks nofollowFlags = unix.O_RDONLY | unix.O_NOFOLLOW

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Network for all nodes of kubernetes cluster (29 hosts) was turned off for a day. Once the network was brought back, we've scaled down a statefulset that mounts a from local volume which on a root drive and then scaled it back up to previous number of replicas. kubelet on 26 out of 29 hosts proceeded with removing all data from subpaths of the pod. In fact, it was ongoing, such that when we'd create a file on a mounted subpath within the container, the file would be cleaned up within a couple of seconds. The issue would continue until volume-subpath was manually unmounted (with command). Ended up removing 3rd of all of our data (we have 3 drives per host and one was wiped) for the whole cluster. Equivalent definition. Note that host path of local volume has to be on root drive: Kubelet logs (filtered out logs from other pods) during the issue: What you expected to happen: Expected no data loss for peristent local volumes. How to reproduce it (as minimally and precisely as possible): Unfortunately was not able to reproduce. But this bug hit 26 out of 29 nodes simultaneously. Need some way to create a case where a pod becomes orphan and "volumes" folder gets somehow removed (similar to issue ). Then there is a code path that can lead to data loss: We are in orphan pod cleanup and directory is missing (not sure why it's printed twice): No error returned and empty volumes list, so we go into this case which proceeds removing pod directory and along with it even thought they might still be mounted (not sure why): checks if device is the same, but we use root drive for local volume, so the device is the same, so it's not a mount point so ok for removing (this is a bug IMHO). Anything else we need to know?: k8s version upgrades went from 1.9.6 -1.11.1 -1.12.1 -1.12.2 -1.13.0 without restarting the pods.

Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OVH, baremetal servers OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Using Ansible tasks that downloads kubelet from and all other required tools. Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Thank you for digging into the code. I think there are a number of things that can be done here: why returned no volumes. why base volume was successfully unmounted. Unmounting the volume should have failed because the subpath bind mount would have kept a reference on it. to also look for the volume-subpaths directory to detect bind mounts.
I am working on a fix for 3 and 4
, do you mind to share us the full kubelet log? You can email me jinxu at Thanks!
why podVolumesExist returned no volumes. From the available log, we can see that Unmount volume and Unmount device succeeded. That's why podVolumeExist will return no volume.
/priority critical-urgent
That's unexpected. The subpath bind mount should have prevented the local volume bind mount from successfully unmounting.
Maybe the fact that the local volume mount is itself a bind mount could be the reason
about to why podVolumesExist returned no volumes. I think we do have an issue here podVolumesExist checks pod volume paths and whether they are mounted. Suppose it should stop cleaning up if there are paths still mounted. However, it only checks normal pod volumes /pod-id/volumes, it does not check subPath /pod-id/volume-subpath directories. We should fix this.
I'm adding a check for volume-subpath directories to address 3)
Jing found that the subpath volume directory uses the outer volume spec name, which is wrong. We made a fix for this in 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16, which changed the on-disk directories to use the inner volume spec name.

had an that in-place node updates were not supported and you would have to drain your pods first. This explains why the volume cleanup proceeded without waiting for the subpath mount cleanup.
We haven't run a version below 1.9.6 on this node.
I've sent the entire kubelet log to your email
to clarify, 1.9.6 uses the old on-disk directory name, so upgrading to anything at least 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16 requires pod draining first.
Right, I remember stumbling upon when debugging this. However, the release notes say: Shouldn't it specify exact version range where the OuterVolumeSpecName was used (1.9.4-1.9.6)? Unless I'm misreading release notes. I think it would be useful to clarify that.
ah you're right. The release notes should say from at least versions 1.7.14, 1.8.9, and 1.9.4
(will update it)
regardless, I still intend to add more safety checks to this orphaned pod cleanup function
From the log provided by the customer, I think we can confirm the reason of this issue is what mentioned By adding the safety checks during pod cleanup will help avoid this issue.


Question: } s := dstrings.Split(sourceName, "-image-") if len(s) != 2 { <del> return nil, fmt.Errorf("sourceName %s wrong, should be pool+"-image-"+imageName", sourceName) </del> <ins> // The mountPath parameter is the volume mount path for a specific pod, its format // is /var/lib/kubelet/pods/{podUID}/volumes/{volumePluginName}/{volumeName}. // mounter.GetDeviceNameFromMount will find the device path(such as /dev/rbd0) by // mountPath first, and then try to find the global device mount path from the mounted // path list of this device. sourceName is extracted from this global device mount path. // mounter.GetDeviceNameFromMount expects the global device mount path conforms to canonical // format: /var/lib/kubelet/plugins/kubernetes.io/rbd/mounts/{pool}-image-{image}. // If this assertion failed, it means that the global device mount path is created by // the deprecated format: /var/lib/kubelet/plugins/kubernetes.io/rbd/rbd/{pool}-image-{image}. // So we will try to check whether this old style global device mount path exist or not. // If existed, extract the sourceName from this old style path, otherwise return an error. glog.V(3).Infof("SourceName %s wrong, fallback to old format", sourceName) sourceName, err = plugin.getDeviceNameFromOldMountPath(mounter, mountPath) if err != nil { return nil, err } s = dstrings.Split(sourceName, "-image-") if len(s) != 2 { return nil, fmt.Errorf("sourceName %s wrong, should be pool+"-image-"+imageName", sourceName) } </ins> } rbdVolume := &v1.Volume{ Name: volumeName,

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: The method for ceph rbd volume plugin generates the path with code as follows: So the device mount path's format will be: , this implementation has two problems: with other volume plugins, as all the other plugins return , see as an example. method of ceph rbd plugin is not working properly, because rbd plugin uses method to find the device mount path(just like all the other plugins do), and method assumes the device mount path's format is (see ). What you expected to happen: The device mount path of rbd plugin should be . Anything else we need to know?: Do we have any reason for this design? If not, I will submit a PR to fix this, the adjusted method maybe look like this: /sig storage
/assign
nice catch There are two ways to fix this: Change mount path. But this will introduce compatibility issues during upgrade. Use , instead of to replace PR welcome
cc
Hi , there is a difference between and : returns the device path, such as , and returns the device mounted path, such as . What we really want is the second one, as we need to find the pool and image name. So we can't just simply replace with . Maybe we can find the mounted path specially for rbd plugin, use , the code looks like this:
thanks let make a forward moving plan: fix and append to the global path keep in . If no path is found, fall back to old path scheme using your code
Nice idea , I've submit a PR for this, PTAL.
I've checked and the reason is that I tested with the old release branch, the has been changed with in mount path, and causes the problem. thanks for your PR
i read again. There is another return path . So in rbd, the missing dir will not make into , but the second return should still give us the dir we need
You are right , indeed has two return path, but the second return still can't meet our needs. Because the parameter of is the mount path for a specific pod, such as , its format is , so will return the volume name. This volume name is returned by (see ), usually is the PV's name or the name specified in , both of these are not .


Question: expectErr:      true, expectedErrMsg: "unable to provision SubPath `no/backsteps/../allowed`: must not contain '..'", }, <ins> "volume doesn't exist": { podVolumes: kubecontainer.VolumeMap{}, container: v1.Container{ VolumeMounts: []v1.VolumeMount{ { MountPath: "/mnt/path3", Name:      "disk", ReadOnly:  true, }, }, }, expectErr:      true, expectedErrMsg: "cannot find volume "disk" to mount into container """, }, "volume mounter is nil": { podVolumes: kubecontainer.VolumeMap{ "disk": kubecontainer.VolumeInfo{}, }, container: v1.Container{ VolumeMounts: []v1.VolumeMount{ { MountPath: "/mnt/path3", Name:      "disk", ReadOnly:  true, }, }, }, expectErr:      true, expectedErrMsg: "cannot find volume "disk" to mount into container """, }, </ins> } for name, tc := range testCases {

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The makeMounts function adding volumes to the volumes list if the volume doesn't exist or mounter doesn't exist. This is potentially dangerous behavior because an application can write to the mount path (if it already exists in the container image) expecting it to be backed by a persistent volume, but in fact, is only backed by the container rootfs, which is ephemeral. We probably have not seen this issue manifest in the wild because the volume not existing or mounter being nil should never happen, and also the application would probably see a path/file does not exist error first. What you expected to happen: If we can't mount all of the user's requested volumes into the container, we should fail to start the container. /assign /sig storage node /cc


Question: } if len(output) == 0 { <del> return 0, fmt.Errorf("can not get image size info %s: %s", b.Image, output) </del> <ins> return 0, fmt.Errorf("can not get image size info %s: %s", b.Image, string(output)) </ins> } <del> // Get the size value string, just between `size ` and ` MB in`, such as `size 1024 MB in 256 objects`. sizeIndex := strings.Index(output, imageSizeStr) divIndex := strings.Index(output, sizeDivStr) if sizeIndex == -1 || divIndex == -1 || divIndex <= sizeIndex+5 { return 0, fmt.Errorf("can not get image size info %s: %s", b.Image, output) } rbdSizeStr := output[sizeIndex+5 : divIndex] rbdSize, err := strconv.Atoi(rbdSizeStr) if err != nil { return 0, fmt.Errorf("can not convert size str: %s to int", rbdSizeStr) } </del> <ins> return getRbdImageSize(output) } </ins> <del> return rbdSize, nil </del> <ins> func getRbdImageSize(output []byte) (int, error) { info := struct { Size int64 `json:"size"` }{} if err := json.Unmarshal(output, &info); err != nil { return 0, fmt.Errorf("parse rbd info output failed: %s, %v", string(output), err) } return int(info.Size / rbdImageSizeUnitMiB), nil </ins> } // rbdStatus runs `rbd status` command to check if there is watcher on the image.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: kubectl create -f ceph- and result is ok now i want to resize pv from 1G to 2G, edit kubectl describe pvc pvc-expand-test What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: ceph storageclass yaml: ceph secret: Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T14:41:50Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration:OS (e.g. from /etc/os-release): CentOS Linux release 7.4.1708 (Core) Kernel (e.g. ): Linux m01 3.10.0-693.el7.x8664 SMP Tue Aug 22 21:09:27 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: ceph version 13.2.2 () mimic (stable) <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
What's the version of your ceph cluster? k8s uses the command to get rbd image's size, and it expects the size unit in the output is . However, in your environment the unit is :
ceph version 13.2.2 () mimic (stable)
It seems that the output format is different between different ceph versions.

I will try to fix it.
/assign
your PR has been merged to master branch, and when will Kubernetes v1.11 fix this bug?
Sorry for the late, I've send cherry-pick PRs:
It seems this change is not working in some cases: I think this is due to two concurrent conditions: 1- ceph client still complain on stderr about not being able to load config file even if there is no need as a key is supplied and the result is (hence the output on stderr of ). This happens not only in hyperkube. 2- it seems we capture both stdout and stderr from the execution of the command (please note the output in the error from ) and then there is then a problem trying to unmarshall the JSON (to be clear, the unuseful warning is on stderr, the json output on stdout). I am able to have a workaround creating empty files for and so that the ceph client does not output anymore but I feel it would be much better to parse only stdout. A reference on the source code for Thanks edit:typo
could you tell where did you put the empty and ceph.keyring files? On the node running kubelet?
Inside the pod of the as the rbd resize is executed by this component.
I had the same problem, I ceph. conf and ceph. Keyring and solved it, but I think k8s got the default user of CEPH before going back to ceph. conf and ceph. Keyring. The following code validates my conjecture: //num 672 Ultimately, I want k8s to get those two files through configurable CEPH parameters. What do you think?
as far as I know you do not need to configure those two files if you use a key. That is still an hypothesis as in the version of Ceph we are using this switch seems to be undocumented: Anyway the warning can be safely ignored and I do not see why we should parse when there is no error
I two files with empty content before, no error; later I a soft connection called and ceph. Keyring (pointing to the ceph configuration file and key ring generated by my cluster), but ceph did not report an error. . I think ceph only needs two files to exist and doesn't care about its contents.

Fortunately, the current measures can solve the problem.
this matches my experience (see above, I created the empty files too and it gives no error) but another possibility is just not parse which I find "cleaner" as there is no error in this case even from the error code, and even what we get is just a warning.
Same issue as here. The warnings about the missing etc files break the JSON output. Adding empty files in the container fixes resizing.
Shouldn't this be reopened until the bug is fixed? Creating the two files also "solves" the issue for us.
/reopen cc
Reopened this issue. <detailsIn response to : /reopen cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
which version of ceph cluster is in use in your setup ?
We are running a ceph cluster with . kube-controller-manager is running which internally uses . Also just to prevent any confusion: I've used to resize the PVC, not rancher.
This also seems related to .
We are running
Are there plans to resolve this issue?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available .

If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Answering myself - apparently it has been fixed in k8s 1.20: