Question: Corefile: | .:53 { errors <del> log stdout </del> <ins> log </ins> health <del> kubernetes {{ pillar['dns_domain'] }} {{ pillar['service_cluster_ip_range'] }} </del> <ins> kubernetes {{ pillar['dns_domain'] }} {{ pillar['service_cluster_ip_range'] }} { pods insecure } </ins> prometheus proxy . /etc/resolv.conf cache 30

Is this a BUG REPORT or FEATURE REQUEST?: BUG What happened: When CoreDNS is deployed by default using feature-gates of kubeadm, it does not log anything when processing a DNS query. That is due to change of log option of CoreDNS in latest version. that version was bumped up in kubeadm, but the manifest was not adapted to the new log option. initial change of 'log stdout' in CoreDNS : version of CoreDNS 1.0.0 bumped into Kubeadm : NOTE: I guess we should also changed manifests for kube-up : see The change in manifest expected is to remove the option "stdout" in the line "log ...". As it is now interpreted as a scope of domain to log. Invalid manifest: Environment: Kubernetes version (use ): 1.9 and MAIN Cloud provider or hardware configuration: any Install tools: kubeadm, kube-up the configmap for coredns kubectl get configmaps -n kube-system coredns -o yaml kubectl replace -f - the coredns deployment kubectl scale deployment --replicas=0 -n kube-system coredns kubectl scale deployment --replicas=1 -n kube-system coredns
/cc /sig network
NOTE: only log is impacted. CoreDNS working as expected.
/cc
I have a question : the initial manifest is done for the image 0.9.10 of CoreDNS. Bumping the version of CoreDNS was done by changing the version in file (see ) which means that the image's version is STILL 0.9.10 in the manifests ... and the corefile config is correct against the version of the manifest. QUESTION: when modifying the , is there a way to upgrade the manifest by program ? I guess at the time the new version number is applied on the manifest ...
thanks for reporting this! I send PR to update the Corefile. I don't think of a way so far. I guess we can't unless there is a way to get the Corefile from some place of upstream (CoreDNS).


Question: }, }, StopError:       nil, <del> ExpectedActions: []string{"get", "list", "get", "update", "watch", "delete"}, </del> <ins> ExpectedActions: []string{"get", "list", "get", "update", "get", "delete"}, </ins> }, { Name: "NoOverlapping",

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite} Previous issues for this test:
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle

30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
I'm pretty sure, those amount of failure are related to: (I don't know why yet).

https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}
https://k8s- Failed: [] Load capacity [Feature:Performance] should be able to handle 30 pods per node {Kubernetes e2e suite}


Question: } // Initialize passes a Kubernetes clientBuilder interface to the cloud provider <del> func (c *Cloud) Initialize(clientBuilder controller.ControllerClientBuilder) {} </del> <ins> func (c *Cloud) Initialize(clientBuilder controller.ControllerClientBuilder) { c.clientBuilder = clientBuilder c.kubeClient = clientBuilder.ClientOrDie("cloud-provider") c.eventBroadcaster = record.NewBroadcaster() c.eventBroadcaster.StartLogging(glog.Infof) c.eventBroadcaster.StartRecordingToSink(&v1core.EventSinkImpl{Interface: v1core.New(c.kubeClient.CoreV1().RESTClient()).Events("")}) c.eventRecorder = c.eventBroadcaster.NewRecorder(scheme.Scheme, v1.EventSource{Component: "aws-cloudprovider"}) } </ins> // Clusters returns the list of clusters. func (c *Cloud) Clusters() (cloudprovider.Clusters, bool) {

On AWS when we call "attach" of a volume then we periodically we check (via API call) if volume has indeed been attached. We run the check in a loop with timeout of 21 minutes. AWS documents that - if a volume is stuck in "attaching" state for too long, the node where volume is being attached is in "dirty" state and must be rebooted to fix the problem. Until node is rebooted, more pods with volumes to same node is only going be stuck in same state (i.e "attaching"). We propose that - if attach operation times out waiting for volume in "attaching" state, we should simply taint the node so as it can not affect other pods and a admin can manually fix the node. /sig aws /sig storage cc


Question: BasePath: cacheDir, TempDir:  filepath.Join(cacheDir, ".diskv-temp"), }) <del> t := httpcache.NewTransport(diskcache.NewWithDiskv(d)) </del> <ins> t := httpcache.NewTransport(&sumDiskCache{disk: d}) </ins> t.Transport = rt return &cacheRoundTripper{rt: t}

We use a lot of CRDs (frequently more than 1,000) in the project. This has historically caused a bunch of problems with client-side API discovery, among other things. At the time of writing the latest releases of kubectl use a discovery client rate limiter with a generous QPS and burst (50 qps, 300 qps burst), so client-side rate limiting doesn't become a problem until the API server is serving more than 300 API groups. Despite rate-limiting no longer coming into effect, we noticed that discovery was very slow - taking 10-20 seconds to complete. After having a few folks at run a little test (see and following comments) we noticed that discovery specifically seems slow on MacOS - it's dramatically faster on Linux. Here's discovery happening on Linux with 1,577 CRDs across 343 API groups in around a second: ` Here's the same test run from MacOS against the exact same EKS cluster, taking 12+ seconds. I expected kubectl discovery on MacOS to be as fast as it was on Linux. an EKS cluster. ~1,500 CRDs across ~350 API groups. the discovery cache in how long the first command you run against the EKS cluster takes to complete Refer to and subsequent comments for more examples of folks reproducing this. <detailsNote that the version of kubectl was the same on Mac and Linux for my tests. </details<detailsAWS - EKS </details<detailsMacOS 12.4, M1 Max Linux as below </details<detailsN/A </details<detailsN/A </details<detailsN/A </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig cli
will show you how many milliseconds was the GET request taken. There are 37 GET requests after was removed. Is there any difference?
I'll add more details ASAP, but we're confident the slow-down is not in the network (i.e. the many GET requests).

Initial profiling indicates it's something to do with writing what is fetched to the discovery cache on disk.
Watch out for anything where writes trigger : there was an issue on macOS where the only way for an app to be confident that a write had completed was to trigger a whole-filesystem sync. Libraries that like data integrity opted for the slower-but-effective approach. I've stopped using macOS so this hint is from a while back, but it might relevant.
Thanks for the tip The issue definitely seems to be in the . When I apply the diff below to switch over to a the slowness disappears and MacOS performance matches that of Linux - about a second for to perform discovery:
It seems to be a problem that would occur only once when establishing the . Should we add it to known issues? Some explain about fsync in MacOs in
Hi, I have no strong opinion here.
You're right, and this happens every 6 hours in the latest versions of kubectl when the cache is invalidated. It's still not an ideal user experience though - you can imagine kubectl 'hanging' for 10-20 seconds the first time you used it each day.
Also note that this should fix a lot of that problem:
has anyone been able to profile kubectl to see what it's doing?
Also note that your CPU is mostly doing nothing, so it's most likely stuck on IO
I think is correct - my bet is we're blocking on , which per uses on MacOS. Per : A mutex profile shows a lot of time waiting for inside , which you can see sets the argument to . I believe this means we're doing an for every HTTP response body we encounter while doing discovery per .
I've confirmed the slowdown is coming from the . Using the following diff: I see MacOS is down to around one second, similar to Linux:
How do folks feel about not explicitly calling for each file in the HTTP cache? Given that it is indeed only a cache I lean toward optimizing for write performance rather than data safety. If we did want to stop calling it seems like the best option would be to update the package of that we use today to make whether it calls with configurable. I'm skeptical we could get this change merged upstream though - the latest change to was three years ago. Another option might be to switch the the package.

I just took a brief shot at this, but it seems like it would require some refactoring to the to handle calling on the underlying database.
I don't care a ton if we drop a cache file coherently. I care a lot if a write fails in a way that leaves a partially written file in a way that pollutes the cache and makes the user have to delete the file from the cache manually. I don't know which of those we're opening ourselves up to by writing without sync. Can we find out?
Presuming we do find that not fsyncing can result in leaving the cache in a broken state (as opposed to a partial state) would falling back to invaliding the cache and recreating it be acceptable?
Little more insight at and Notably the second link mentions:
I dug into this a bit - dumping state here before I sign off for the day. First, note that the discovery client uses two layers of cache - a "discovery cache" and an "HTTP cache". The former appears to be backed by the latter. My -fu is not strong enough to figure out why we do this. The best I could find is , which refers to "restoring the old cached client behavior" and appears to (re?)introduce the caching HTTP transport. Perhaps or know why this is? Only the HTTP cache is explicitly calling . The discovery cache just writes the files and calls , so it's already subject to the risks of not using . Per that risk is that despite the write having been flushed from the OS to the disk controller (e.g. by successfully calling ): My understanding of the and code that powers the HTTP cache is: If a cache file read returns an error we just act as if the value isn't in cache. If a cache file read returns garbled data that can't parse as an http response would fall back to its underlying (i.e. make a call to the API server) and will happily cache over the top of an existing (potentially garbled) file. The caches will be invalidated after six hours. So the failure mode that wouldn't be covered if we stopped calling is that if power were cut to the drive before its controller could sync down to persistent storage and_ this resulted in a file that was garbled just right such that it appeared to be a valid HTTP response but contained an invalid body (i.e.

not real discovery data) someone would have to either manually remove their cache, or wait six hours for the cache to be automatically invalidated. It seems like we could probably make the discovery client force an uncached request through the caching HTTP transport if it encountered what appeared to be cached, invalid discovery data, but I do wonder how far we really need to go to ensure that this cache is never corrupted.
For either of these caches, there are techniques (checksums and the like) where we can allow the app to see whether or not it is looking at a coherent cache document or something that might not be correct. Whenever a cache entry doesn't look valid, the code can asset that the date is current and the entry checksum is OK. If the entry turns out to be invalid or stale, remove it and then act as if we got a read error. That should cover any kind of unsynchronised write failure, I think. For locking the cache we also don't need to ensure sync-to-disk, so long as all running processes see a consistent view of locking.
Agreed. My question is whether we need to add such techniques to this particular cache (or caches) before we'll feel comfortable removing the fsync. Removing the fsync from the HTTP cache would presumably bring it back down to the same level of guarantee as the discovery cache. My guess would be that the fact we were ever explicitly fsyncing each cache value in the HTTP cache was a happy accident rather than something that was explicitly a desirable quality for the HTTP cache.
It seemed like writing our own httpcache cache backend was the best way to go here, so I went ahead and one that uses checksums rather than syncs per the above PR.


Question: effect: "NoSchedule" containers: - name: kube-proxy <del> image: {{pillar['kube_docker_registry']}}/kube-proxy:{{pillar['kube-proxy_docker_tag']}} </del> <ins> image: {{pillar['kube_docker_registry']}}/kube-proxy-amd64:{{pillar['kube-proxy_docker_tag']}} </ins> resources: requests: cpu: {{ cpurequest }}

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-alpha-features Which test(s) are failing: BeforeSuite Since when has it been failing: Failing since 7/18 The failure seems to have been introduced in is the only PR within that window that modifies any cluster configuration. This PR is related to Testgrid link: Reason for failure: kube-proxy is stuck in a pending state /milestone v1.16 /priority critical-urgent /kind failing-test /sig testing /sig cluster-lifecycle /cc
almost definitely related to the image change E0725 05:54:40. 1368 ] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: manifest for not found
/assign
hmm, we should watch to ensure that actually fixed it /reopen
Reopened this issue. <detailsIn response to : hmm, we should watch to ensure that actually fixed it /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Cluster came up in the most recent run, however it looks like tests may fail (it's not done yet). That should be a new issue though. /close
Closing this issue. <detailsIn response to : Cluster came up in the most recent run, however it looks like tests may fail (it's not done yet). That should be a new issue though. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: "//pkg/client/clientset_generated/internalclientset:go_default_library", "//pkg/client/informers/informers_generated/internalversion:go_default_library", "//pkg/cloudprovider:go_default_library", <del> "//pkg/cloudprovider/providers:go_default_library", </del> "//pkg/controller/serviceaccount:go_default_library", "//pkg/generated/openapi:go_default_library", "//pkg/kubeapiserver:go_default_library",

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The current help message from kube-apiserver is wrong. It shows only the 'NamespaceLifecycle' plug-in is available, and the default value is . Users cannot know what are the available plug-ins, and they get confused where comes from. What you expected to happen: The correct output should be this: How to reproduce it (as minimally and precisely as possible): Run using master or at least 1.7.2. Anything else we need to know?: This is a regression introduced when refactoring code into generic server code. Environment: Kubernetes version (use ): 1.7.2+ ? Cloud provider or hardware configuration**: Any OS (e.g. from /etc/os-release): Any Kernel (e.g. ): Any Install tools: Any Others: None
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Yes, it is really exist. If no guys works on this, i will pick it up.
Thanks for confirming this. I'm already working on it. Will submit pr in minutes.
cool. :)


Question: kube-scheduler,ppc64le/busybox kube-proxy,gcr.io/google_containers/debian-iptables-ppc64le:v3 federation-apiserver,ppc64le/busybox <del> federation-controller-manager,ppc64lebusybox </del> <ins> federation-controller-manager,ppc64le/busybox </ins> );; esac

disabled PPC builds due to the following error: Additional comments here
cc
The issue on PowerPC is both for cross and native-builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
Have you tried to upgrade to a newer version? It might or might not help... You might also try to look at what in the gcc linker differs from for example, where it doesn't fail
We are looking into this and it might take some more time. In the meanwhile is it possible to remove hyperkube from the ppc64le builds? All other binaries are building perfectly fine. We can resume the hyperkube for ppc64le build as soon as this issue resolves. cc:
Ah, this is only partially fixed. Reopening...
The error occurs because the hyperkube binary is very large and when building the binary, golang is generating a single go.o file with a single text section that is larger than 2^26 on ppc64le. When there is a call instruction within that section where the call target offset is beyond what can fit in the instruction the relocation truncation error will be generated by the GNU linker. When there are multiple, smaller text sections sent to the GNU linker for linking, it can generate appropriate call instructions even the call target is beyond that limit. I'm working on a fix to do that but hitting other issues as a result which I'm working through. I suspect that this same error will be hit at some point on arm64 unless their linker has another way to handle this situation. On ppc64le we are very close to the limit and from build to build it varies so that some days it will build and others it won't.
Interesting. How does amd64 handle it?
I don't know for sure how arm64 handles it because I don't have access to a machine that is set up to build for arm64. I suspect their text section size is not as large as it is on ppc64le so they haven't hit the limit yet.
Ok, you may easily see what's going on for arm64 by investigating what does in kube-cross.

But I asked about what amd64 (x86_64) does to prevent this :) (I'm assuming this will never happen to amd64)
Sorry I misread your question about amd64... The call instruction in amd64 has a 32 bit field instead of a 24 bit field in ppc64le for the call target address. That means they wouldn't hit this error when linking hyperkube until its text section was larger than 2^32. If you search on "relocation truncated to fit" you will see that this error has happened in the past on x86 with other programming languages.
Once a fix is ready, how does it need to be provided?
I saw https://go- probably won't make 1.7, and that's sad. We would just upgrade golang version I think, not really sure what you're asking for
I assume you only use official golang versions or those provided with distros for your builds? Are you doing all your builds as cross builds from a golang on an x86?
Official versions, yes. We're using the docker image. All official builds are built on amd64 hosts.
Where are you getting this docker image from? Will you be moving to golang:1.7 when it is released?
We're using the official images, and will pick up as soon as it is released for testing I think. Then if the testing and scalability teams think it's safe to use, we'll switch to 1.7 BTW, where is your patch for this? Can you link to it?
is also broken right now - see .
Hi, I'm sorry I missed your last question to me. Here is the CL for my patch for the PPC64 relocations truncation error. https://go- The error with kubemark is also size related but somewhat different because that one is being built as static. Just curious, why are some built static and some non-static?
just to tie everything together, disabled linux/ppc64le compilation entirely. hopefully by go1.8 we can reenable it.
Well, we've hit 1.8 I think with Want to link to any other relevant issues to kinda tie this one together, and close this if it seems appropriate?
yeah, ppc64le builds have been re-enabled for k8s 1.6.

The issue on PowerPC is both for cross and native builds. The issue appears to be with the toolchain (golang) and how the linking is being done. The object sections have grown too large to link. We are looking at it cc
this issue is quite old. Can you provide more context to your comment? Specifically, more detail to help me reproduce. I was unable to find anything related to ppc64le on .


Question: } // UpdateTCPLoadBalancer is an implementation of TCPLoadBalancer.UpdateTCPLoadBalancer. <del> func (gce *GCECloud) UpdateTCPLoadBalancer(name, region string, hosts []string) error { </del> <ins> func (gce *GCECloud) UpdateTCPLoadBalancer(origName, region string, hosts []string) error { name := normalizeName(origName) </ins> pool, err := gce.service.TargetPools.Get(gce.projectID, region, name).Do() if err != nil { return err

GCE has a length limit of 63 characters for its load balancer components (target pools and forwarding rules), and AWS has a length limit of 32 characters. Our current load balancer name construction of clustername-namespace-servicename can exceed both limits, especially AWS's. We should use a shorter name construction method less likely to hit these limits.
Thanks for filing this Alex. My suggestion would be to use a simple UID, as per utils.NewUID(). We'll need to store the mapping from {clustername, namespace, servicename} -UID somewhere stable like etcd. Using anything based directly on {cluster, namespace, name}, for example a stable hash of each, has the problem that the aforementioned tuple is not unique over time. This raises the possibility of e.g. the following scenario: creates mycluster/mynamespace/myservicename, which creates external load balancer named stablehash(mycluster,mynamespace,myservicename) deletes mycluster/mynamespace/myservicename, but at the time MyCloudProvider is down, so kubernetes can't delete ELB named stablehash(mycluster,mynamespace,myservicename). Kubernetes keeps trying in the background to delete the ELB. the mean time user creates a new service named the same as her old one mycluster/mynamespace/myservicename. Kubernetes tries to create an associated ELB named stablehash(mycluster,mynamespace,myservicename), but this fails, because the ELB exists already due to 2 above. ... etc. Badness ensues.
Assigning to Chao Xu, as this makes for a good starter bug.
Hmmmm.... can't seem to assign to caesarxuchao yet. Not sure why.
He has to have write privileges on the repo before issues/PRs can be assigned to him.
But we have many contributors with issues assigned to them that are not in the kubernetes-write group (that only has 48 members). I must be missing something. How do I grant him write privileges on the repo?
We do? That's not how I understand it to work: If you want to add him, go .
has been because the load balancer name changes if the service's UID changes (e.g. if it's deleted and recreated).

Think we can change this? A hash of cluster name, namespace, and service name should get around it, but there may be better ideas as well.
Sure. I'm not confident that I can fix this quickly. If it's urgent, maybe you can assign some one more experienced to fix it? One question: I guess the ELB's name should stay unchanged as long as the "zone+namespace+service name" doesn't change, am I right?
I think I'm the guilty party here. I was unaware that when we create a new service with the same name as a previously existing service, that it needed to keep the same load balancer name and IP. This seems broken to me, but I may be missing the larger context. We can quite safely roll back the change while we decide on the best alternative solution. Q , Chao Xu wrote:
I think the real breakage is that we don't have any way to claim an LB-IP at a larger lifespan than a single service. Alex, can you go back to OP and get a sense of what's happening there to trigger this? , Quinton Hoole wrote:
Will do. Even if the name is kept consistent as it was before, with the way everything currently works the IP won't be consistent.
yeah, I'm not positive that the name has to be consistent. What we need is a way to claim the same public IP address, and we already have that (at least for GCE) Assigning to me, and I'll validate the length requirements, make sure public ip assignment works for AWS and close out this issue.
Note there are two separate things here: (1) a purely-name-based construction can exceed length limits given a single service and (2) some form of across-service consistency is apparently needed. (1) is trivially solvable by either always using a hash or alternatively using a two-stage process where you generate a desired name and only hashify if it's longer than the length limits by truncating it and appending a hash of said name.
AWS's load balancer hasn't been merged yet, so there's not much to check out there. I'm going to be reducing the churn of load balancers later today, which should help prevent IP address changes for services that are modified.

I think we need to be able to figure out which Service an LB is attached to from the Service, so that LBs can't get orphaned. regarding being able to claim an LB IP. I don't think that is covered in revamp... , Brendan Burns wrote:
I think that we need to roll back and think this issue through a bit better. See comment on that PR.


Question: informerFactory.Core().V1().Nodes(), clientset, ) <del> manager.eventRecorder = record.NewFakeRecorder(100) </del> <ins> fakeRecorder := record.NewFakeRecorder(100) manager.eventRecorder = fakeRecorder </ins> manager.podStoreSynced = alwaysReady manager.nodeStoreSynced = alwaysReady

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): DaemonSet FailedPlacement Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): OL7.3 Kernel (e.g. ): 4.1.12-94.2.1.el7uek.x86_64 Install tools: Others: What happened: I have a DaemonSet that is only deployed to a subset of nodes in the cluster based on a . I'm seeing events fired with reason for the daemonset pods on nodes that do not fit the selector. What you expected to happen: These events should never fire because they do not meet the node selectors in the pod spec. How to reproduce it (as minimally and precisely as possible): Create a cluster with different sized nodes. E.g. smaller masters than workers. Then deploy the daemonset only to the workers. You will see events triggered for the master nodes that don't meet the requirements. Anything else we need to know:
DaemonSet controller will run predicates to check if node should run daemon pod. In the progress of predicates, it will check if pod fits resources first before other predicates such as PodSelectorMatches. /cc
I think this bug appeared when critical pods were introduced and order of checking predicates changed. The event is useless for daemonsets in current situation :/ I'm not sure if there is a clean way to fix it.
I feel like a clean solution would be to look at the predicate reasons twice. The first time you look at the reasons and keep track if any are not intentional. If all of the reasons are intentional then skip checking the reasons.
actually just changing the order of checks in GeneralPredicates may just work. I'm testing the fix now.
Ok, GeneralPredicates returns all reasons. Error that pod doesn't match the node is also returned but because it's "intentional" there is no event for it.

The event "Node didn't have enough resource" should be sent only when there are no other "intentional" reasons.
Yes, I think maybe we need to check reason list twice: check is in tail of predicate list.
/cc
any PR for this? If not, I can help
I will have some time to work on it next week. If you have a spare time you can create a PR.
/assign


Question: // always try to update the sched.nextStartNodeIndex regardless of whether an error has occurred // this is helpful to make sure that all the nodes have a chance to be searched processedNodes := len(feasibleNodes) + len(diagnosis.NodeToStatusMap) <del> sched.nextStartNodeIndex = (sched.nextStartNodeIndex + processedNodes) % len(nodes) </del> <ins> sched.nextStartNodeIndex = (sched.nextStartNodeIndex + processedNodes) % len(allNodes) </ins> if err != nil { return nil, diagnosis, err }

On Kubernetes v1.30.0 (and v1.30.1), can crash with the following panic if a pod is defined in a certain way: The crash happens because the is in certain cases. should not crash, on v1.29.4 it doesn't happen. On v1.29.4 the is just printing these error logs, but doesn't crash: Create a pod like this: The important part is that the affinity doesn't match a real/valid node. Once this pod is being processed by it will crash and continue to do so until the pod is deleted. This issue was triggered during the rotation of control-plane nodes. In our setup we update the control plane by scaling from 1 to 2 instances. Once both a ready, the old is being terminated via EC2 API. At this stage the kube-controller-manager sometimes_ manages to create a replacement daemonset pod once the old node is being deleted. This results in a pod targeting a no longer existing node via as illustrated in the example above. For some reason, when the is crashing the doesn't delete the extra/invalid daemonset pod. Not sure if this is another issue or it has always happened in our setup, but only v1.30 makes crash which causes an actual issue. <details</details<detailsWe run a custom Kubernetes setup on AWS EC2 (not EKS). </details<details</details<detailsCustom tooling: </details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig scheduling
cc /priority important-soon
can you take a look? I believe all the latest patch releases are affected too, because of
cc
/priority critical-urgent
This crash happens when preFilter plugins filtered out all nodes ... If preFilter filtered out some nodes, here len(nodes) will be a subset of allNodes. Why are we using this subset at first? I think should works fine?

(seems logical to me)
I think so. In general, we just want to try a different set of nodes. For the case of Daemonsets, it doesn't really matter, as we will just test one node.
/assign Successfully catched this with a unit test. I'll file a small pr with fix and a newly UT.
I am having the same issue with 1.28.10 on a clean new cluster created with kops 1.28.5: The pod that it's trying to create is pending with the following messages: (just for the sake of being indexed for those who will search for the same error to find this github issue)
Fix is on the way: Just waiting for reviews from approver. After this pr got merged, I'll do cherry-picks for v1.28(1.27?)~1.30
cherry-picks:
Thanks for fixing this so fast. How do I pull this new code? I see that your PR was merged, but I don't see a new release to pull? looking for the 1.30
This change will be included in the upcoming patch releases, scheduled for 2024-06-11. You can check out the for more details.
I'm seeing the issue after upgrade v1.26.11-v1.27.14 Should I search for a broken pod? Or what else can be causing this?
Yes, a broken pod could be causing this. You can also try downgrading to v1.27.13
Phew, manually changing the scheduler version to 1.27.13 fixed the issue and showed the broken pod in logs. Thanks!


Question: return pod, nil } <del> var simpleSubdomainSafeEncoding = base64.NewEncoding("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ012345678900") </del> <ins> var simpleSubdomainSafeEncoding = base32.NewEncoding("0123456789abcdefghijklmnopqrstuv") var unsafeDNSLabelReplacement = regexp.MustCompile("[^a-z0-9]+") </ins> <del> // simpleSubdomainSafeHash generates a compact hash of the input that uses characters // only in the range a-zA-Z0-9, making it suitable for DNS subdomain labels func simpleSubdomainSafeHash(s string) string { </del> <ins> // simpleSubdomainSafeHash generates a pod name for the given path that is // suitable as a subdomain label. func simpleSubdomainSafeHash(path string) string { name := strings.ToLower(filepath.Base(path)) name = unsafeDNSLabelReplacement.ReplaceAllString(name, "") </ins> hasher := sha1.New() <del> hasher.Write([]byte(s)) </del> <ins> hasher.Write([]byte(path)) </ins> sha := simpleSubdomainSafeEncoding.EncodeToString(hasher.Sum(nil)) <del> if len(sha) > 20 { sha = sha[:20] } return sha </del> <ins> return fmt.Sprintf("%.15s%.30s", name, sha) </ins> }

The hash function generates names with upper-case letters, but the validation only allows lower-case. First, we clearly need a test for this. Second, we need to fix it. Should we allow upper-case characters (which the RFC does allow for DNS labels) or should we fix the hash? Allowing upper-case seems attractive, except that the RFC also says that case doesn't matter, but should be preserved "wherever possible". So we really have three choices, I see. a) only allow lower-case b) allow upper-case and always use caseless comparisons everywhere c) allow upper-case, but normalize to lower-case since it was your hash code
Re: test - sounds like a config source -config integration test, which I'll add. Re: hash - would argue we should fix the hash to generate lowercase for now. I think, but will double check, that the collision space is still improbably low for 63 lowercase chars. I can do that shortly.
And a validate test on all sources.
Lowercase would require base32, rather than base64, no? On Aug 3, 2014 8:20 AM, "Clayton Coleman" wrote:
Yeah, although I don't believe there's a difference to the outcome between base32 and base64(lowercase) given an appropriate hash function.
Well, base32 is reversible and a modified base64is not :) On Aug 3, 2014 9:00 AM, "Clayton Coleman" wrote:
Remind me again why reversibility is important? We discussed it in another issue but I can't find the context.
I'm going to change the generated name to which even with 2^8 files per host and a thousand hosts should be well below 0.% chance of collision
Its only important if we want to use it. We don't AFAIK. On Aug 3, 2014 9:34 AM, "Clayton Coleman" wrote:
Fix makes them unique, somewhat traceable (first 15 safe chars of the name), and generates base32 output.


Question: expectedRule.LoadBalancingRulePropertiesFormat.IdleTimeoutInMinutes = lbIdleTimeout } <del> // we didn't construct the probe objects for UDP or SCTP because they're not used/needed/allowed if protocol != v1.ProtocolUDP && protocol != v1.ProtocolSCTP { </del> <ins> // we didn't construct the probe objects for UDP or SCTP because they're not allowed on Azure. // However, when externalTrafficPolicy is Local, Kubernetes HTTP health check would be used for probing. if servicehelpers.NeedsHealthCheck(service) || (protocol != v1.ProtocolUDP && protocol != v1.ProtocolSCTP) { </ins> expectedRule.Probe = &network.SubResource{ ID: to.StringPtr(az.getLoadBalancerProbeID(lbName, lbRuleName)), }

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Refer , health probes are not associated for UDP services on Azure, hence source IPs are not preserved for UDP services on Azure. What you expected to happen: When is set to for UDP services, source IPs should be preserved and the health probe should be associated with UDP load balancer rule. Per the , the health probe only supports TCP, HTTP and HTTPS. UDP is not supported as the probing protocol, hence we can’t enable the health probes for UDP services. However, when the externalTrafficPolicy is Local, Kubernetes would open an HTTP endpoint for UDP services, hence it’s actually supported on Azure. So, should associate the probes on externalTrafficPolicy=Local cases. How to reproduce it (as minimally and precisely as possible): Create a UDP service with : Then check the source IPs. Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /sig cloud-provider /area provider/azure


Question: return refCount, nil } <del> // make a directory like /var/lib/kubelet/plugins/kubernetes.io/iscsi/portal-some_iqn-lun-lun_id func makePDNameInternal(host volume.VolumeHost, portal string, iqn string, lun string) string { return path.Join(host.GetPluginDir(iscsiPluginName), portal+"-"+iqn+"-lun-"+lun) </del> <ins> // make a directory like /var/lib/kubelet/plugins/kubernetes.io/iscsi/iface_name/portal-some_iqn-lun-lun_id func makePDNameInternal(host volume.VolumeHost, portal string, iqn string, lun string, iface string) string { return path.Join(host.GetPluginDir(iscsiPluginName), "iface-"+iface, portal+"-"+iqn+"-lun-"+lun) </ins> } type ISCSIUtil struct{} func (util *ISCSIUtil) MakeGlobalPDName(iscsi iscsiDisk) string { <del> return makePDNameInternal(iscsi.plugin.host, iscsi.portals[0], iscsi.iqn, iscsi.lun) </del> <ins> return makePDNameInternal(iscsi.plugin.host, iscsi.portals[0], iscsi.iqn, iscsi.lun, iscsi.iface) </ins> } func (util *ISCSIUtil) AttachDisk(b iscsiDiskMounter) error { var devicePath string var devicePaths []string var iscsiTransport string out, err := b.plugin.execCommand("iscsiadm", []string{"-m", "iface", "-I", b.iface, "-o", "show"}) if err != nil { glog.Errorf("iscsi: could not read iface %s error: %s", b.iface, string(out))

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): iscsi session logout Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: Libvirt qemu VM, 4 CPU, 8Gig RAM OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS (Xenial Xerus) Kernel (e.g. ): Linux localhost 4.4.0-62-generic -Ubuntu SMP Wed Jan 18 14:10:15 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm Others: What happened: If you have multiple pods on the same node using PVCs that are backed by iSCSI LUNs on the same external system over a single iSCSI session (same IQN but different LUN numbers), then deleting one of the pods causes the iSCSI session to terminate for all pods on that node that are using it. For example, here and are nginx pods each connected to a different PVC, which are backed by LUNs residing on the same NetApp storage system. These share a single iSCSI session, differing only by LUN number. Once pod2 is deleted, all the connected LUNs are no longer available.

[6:0:0:1] disk NETAPP LUN C-Mode 8300 /dev/sdc What you expected to happen: I expect that in-use iSCSI sessions are not terminated How to reproduce it (as minimally and precisely as possible): 2 PVs and PVCs that are backed by a single storage system that aggrates iSCSI connections into a single session. 2 pods that consume those PVCs one of the pods Anything else we need to know:
Could someone from please take a look? Thanks!
looking...
It looks like the problem is in line 86 of , in the function. The check evaluates whether a given mount path starts with the prefix (protocol + target portal + target IQN) for the LUN; this will always evaluate to false in most set-ups, since each path will look something like . Changing to appears to result in the correct behavior.
The problem is that the mount device shown in /proc/mounts are canonicalized, and thus when searching device prefix, iscsi volume failed to find a match. If we can use mount option, this the device will show up in /proc/mounts. But this is not a reliable fix since there are probable mpio paths too.
If I'm reading the code correctly, the function doesn't look at the actual device that's backing the mount point; it looks at the mount path (i.e., the second token in /proc/mounts, rather than the first). Thus, the change I proposed (HasPrefix -Contains) appears to work with some very basic testing. That said, I've not done anything with multipathing, so there may be something I'm missing; in addition, I suppose there's a chance that Contains may trigger false positives.
I see your point, yes that sounds right to me.
I think in this case, mpio device should work too (since the mount path constructed in the same way). I see you have yet another iscsi issue open, I am good with rescan. Can you create a pr to fix them?
I should be able create a PR for this issue; the LUN rescan PR will probably get handled by someone else on my team. Is the fix itself sufficient, or does it require tests?
thanks, the rescan fix doesn't have to unit test. The mount path fix needs also a unit test fix here

BUG REPORT Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:57:25Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-01-12T04:52:34Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"linux/amd64"} Environment: Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7; reproduced in Ubuntu 14.0.4.2. Kernel (e.g. ): Linux centos7 3.10.0-514.6.1.el7.x8664 SMP Wed Jan 18 13:06:36 UTC 2017 x8664 x8664 x8664 GNU/Linux; kernel 3.16.0-77-generic What happened: On a host where an iSCSI LUN from a given target is mounted in a pod, a user who creates a new LUN on the same target and attempts to create a pod that mounts the new LUN will fail to create the pod. Specifically, depending on the operating system, the pod will hang in the ContainerCreating stage with either a "Could not attach disk: Timeout after 10s" error (CentOS 7) or an error about establishing a session when one already exists (Ubuntu 14). What you expected to happen: The pod should successfully mount the LUN. How to reproduce it (as minimally and precisely as possible): Given a working iSCSI target and a single node Kubernetes cluster: a LUN on the iSCSI target a pod that uses the LUN from step 1 (either natively or through a PV/PVC) a new LUN on the iSCSI target (note that the target portal and IQN must be identical). a pod that uses the LUN from step 3. The pod created in step 4 will fail to start.

Anything else we need to know: As far as I can tell, this is due to a bug with the Kubernetes iSCSI implementation. The AttachDisk() function in first checks if the LUN exists in /dev/disk/by-path/; if it doesn't, it attempts to log in to the target and recheck for the LUN's existence (lines 123-138 in the 1.5 branch). In this case, though, the session already exists, as it was created with the pod in step 2. In Ubuntu, attempting to log in a second time triggers an error, which gets reported in the pod description. In CentOS 7, this gets treated as a NOP; however, the new LUN will never become visible, and thus, the pod will fail to start. This can be solved manually by logging into the host and running . It should be possible to fix it in Kubernetes by adding a check in AttachDisk() to see if a session already exists for the target portal and IQN and, if so, doing a rescan of the session.
Could someone from please help with triage? Thanks!


Question: "k8s.io/apimachinery/pkg/watch" "k8s.io/cli-runtime/pkg/genericclioptions" "k8s.io/cli-runtime/pkg/resource" <del> "k8s.io/client-go/kubernetes" </del> corev1client "k8s.io/client-go/kubernetes/typed/core/v1" "k8s.io/client-go/tools/cache" watchtools "k8s.io/client-go/tools/watch"

The command provides a flag but this doesn't do anything. most likely the flag shouldn't exist. I have no idea how kustomize could be useful here to start a single pod. This creates a pod, and shows that is ignored: No response <details</details<detailsAzure, K3OS</details<detailsOSX 11.6.1 (Big Sur) </detailsNo response No response No response
/sig cli
This comes from the inclusion of DeleteFlags: brings in FilenameFlags: has Kustomize, resulting in the flag: I suspect there are many flags from DeleteFlags that don't really apply to . : Not sure what this would do : seems to be ignored and : doesn't seem to be used : doesn't seem to be used : also probably doesn't apply There might be others as well...
It seems to delete object which created by command.
I agree, As far as I can see, , and have this deleteFlags exposure.
/assign
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted


Question: // barrierOp defines an op that can be used to wait until all scheduled pods of // one or many namespaces have been bound to nodes. This is useful when pods <del> // were scheduled with SkipWaitToCompletion set to true. A barrierOp is added // at the end of each each workload automatically. </del> <ins> // were scheduled with SkipWaitToCompletion set to true. </ins> type barrierOp struct { // Must be "barrier". Opcode string

is a very useful option in scheduler perf test framework to define an operation without waiting for it to finish. However, the current testing logic still waits for all the pods in this operation to be schedulable, in the last step: the number for Pods in this operation to be schedulable This can be problematic as some case just want to generate Pods (for example, to test internal queuing or sorting behavior/latency). The following test command times out as those large-cpu Pods are expected to not fit on any node: So, given is commented as: I'm proposing to relax the semantics to not wait for the Pods to be schedulable if the operation is specified as . /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign WDYT?


Question: credentialprovider.RegisterCredentialProvider("google-dockercfg", &credentialprovider.CachingDockerConfigProvider{ Provider: &dockerConfigKeyProvider{ <del> metadataProvider{Client: http.DefaultClient}, </del> <ins> metadataProvider{Client: &http.Client{Timeout: 10 * time.Second}}, </ins> }, Lifetime: 60 * time.Second, })

http://pr-
Something holding the port already?
The previous masterless case just barely passed with a 12.5 second timeout (25 tries at 0.5 second interval) I think we just need to increase the timeout.
I randomly sampled 5 green builds and they all have this test (starting kubelet in masterful/less mode) finished in 3s or less. Judging for the successful masterless mode startup before the failure, kubelet spent the most time trying to register credential providers (a 10s gap in the log). The same roughly applies to the failed "masterful mode", based on where the log message stopped. From , the registering process basically tries to access the metadata url to see if the credential provider is enabled. I think this may just be the slow response from the metadata server.
I am wondering even if this issue might be related to a slow response from the metadata server, why it does not show "Registering credential provider: .dockercfg" always in failed logs?
If there is a possibility that ReadUrl in may take longer than expected, would it be preferred to use a Timeout (0) with the http client to not get stuck longer than expected?
I am unsure why it didn't show that, but even in the passing round we see a 10 second gap, which almost maxed out the timeout. I think adding a timeout is reasonable, but I am not sure what value is reasonable given that the previous request took 10s. do you have any idea?
Even I am not sure exactly what should be a reasonable value. My guess would be that if in normal cases, average response time is x seconds (perhaps a guestimate could be computed based on "normal" successful tests), then x could be a starting point, or otherwise another option is choosing something randomly (like 5s). And I think that if at some point, the server is responding in 10s, and even though the test passes, its not normal and something is going wrong.
uses a 10 second timeout, but the request will be retried. Given that we don't retry this request, a short timeout is worrisome. Maybe we should add both retry and timeout in this case.
SGTM.
I reopen this one since pr was reverted.

Noticed a similar hang in: The logs are a bit different from those observed in
if kubelet couldn't start up properly, the test itself should've timed out and logged "Timed out waiting for kubelet to answer at http://127.0.0.1:/healthz; tried 25 waiting 0.5 between each". I am not sure if this failure is related.
failure is not relevant to this issue. The most recent report of such flake was over a week ago. I think the cause is the slow metadata url response in GCE, and that should've been fixed by the recent rollout. I am going to close this issue. Feel free to reopen if there is another occurrence.


Question: image: e2e-node-containervm-v20161208-image # docker 1.11.2 project: kubernetes-node-e2e-images gci: <del> image_regex: cos-beta-59-9460-20-0 # docker 1.11.2 </del> <ins> image_regex: cos-stable-59-9460-60-0 # docker 1.11.2 </ins> project: cos-cloud metadata: "user-data<test/e2e_node/jenkins/gci-init-gpu.yaml,gci-update-strategy=update_disabled" resources: accelerators: - type: nvidia-tesla-k80 count: 2

xref: /sig node /area test
I am working on this.
Can you update this with the types of tests that you are adding? Is this similar to ?
No, this is for running the existing GPU e2e tests on GKE which are currently running on only GCE.
The referenced PRs appear to have merged, closing.


Question: expectNoEvent(watchB, watch.Added, testPodA) testPodA, err = updatePod(f, testPodA.GetName(), func(p *v1.Pod) { <del> p.Spec.ActiveDeadlineSeconds = int64ptr(10) </del> <ins> p.ObjectMeta.Labels["mutation"] = "1" </ins> }) Expect(err).NotTo(HaveOccurred()) expectEvent(watchA, watch.Modified, testPodA)

https://k8s- Please work on finding a fix for the flake. /priority failing-test /priority important-soon /sig api-machinery /kind flake
/assign
I think this was fixed by
we had a failure on 3/28 (after the above fix merged). https://k8s- Looking at the logs this seems to be a different error than the one fixed by . Can you confirm plz?
Hmm that is different, and I will look into the reason.
There hasn't been a failure since 3/28, So while the test might have flakes it is definitely less flaky than 12 failures per week now
Yes the latest runs look good. Please close the issue
/close


Question: return nil, fmt.Errorf("failed to find loaded kernel modules: %v", err) } <ins> builtinModsFilePath := fmt.Sprintf("/lib/modules/%s/modules.builtin", kernelVersionStr) b, err := ioutil.ReadFile(builtinModsFilePath) if err != nil { klog.Warningf("Failed to read file %s with error %v. You can ignore this message when kube-proxy is running inside container without mounting /lib/modules", builtinModsFilePath, err) } for _, module := range ipvsModules { if match, _ := regexp.Match(module+".ko", b); match { bmods = append(bmods, module) } else { // Try to load the required IPVS kernel modules if not built in err := handle.executor.Command("modprobe", "--", module).Run() if err != nil { klog.Warningf("Failed to load kernel module %v with modprobe. "+ "You can ignore this message when kube-proxy is running inside container without mounting /lib/modules", module) } } } </ins> return append(mods, bmods...), nil }

<Reopened this valid and in-demand bug after the auto-bot expired it/kind bug /sig network /area kube-proxy What happened: Building a fully static Linux kernel with kubeproxy's desired IPVS (Virtual Server) modules linked statically: CONFIGIPVS=y CONFIGIPVSRR=y CONFIGIPVSWRR=y CONFIGIPVSSH=y CONFIGNFCONNTRACKIPV4=y Causes kubeproxy to believe that these features don't exist because it can't find modules for them: What you expected to happen: If Kubeproxy's desired kernel features are linked statically in the kernel, kubeproxy should carry on and not attempt and throw error messages about modules not being found. Under the hood, kubeproxy should simply check for the existence of the desired kernel features in the /proc tree (ie: , and others) before attempting to look for and load modules. If, after module loads fails, then throw the error messages and switch to iptables. How to reproduce it (as minimally and precisely as possible): and extract the current stable kernel source version (today it's 4.18.9) from your existing running into the source tree as and answer any questions to switch to a fully static kernel and ensure the following exist, and are not : and deselect (box is empty) your boot loader's config () or () depending on your distribution. and select your new kernel with the kubeproxy modules linked statically. kubeproxy and watch it throw the above errors about not finding the modules. Anything else we need to know?: Fully static kernels will not generate so inspecting that file will fail - as well in the kubeproxy source. The gold-standard is to simply check for the features in the active tree first, and consider those as "built-in". The next best options is checking for statements, which tells you if they're built-in. Note that files are almost universally installed by OS maintainers. And finally, you could try checking , which is a compressed version of the externally held .config text file, in a similar manner noted above, however it's up to the maintainer to build that feature into the kernel (so it might not exist).

For reference, here's the list of modules kubeproxy wants: Here we see kube_proxy's errant module-centric code that needs to be made compatible w/ static kernels: Environment: Kubernetes version (use ): 1.11.3 Cloud provider or hardware configuration: On premise OS (e.g. from /etc/os-release): Leap 15.0 Kernel (e.g. ): 4.18.9 statically linked Install tools: gcc 8.1 Others: <reopened as it appears still unfixed and in-demandSee how far the team got on it before it was auto-closed here:
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
Hi is the issue just the log messages, or do the kube-proxy features actually fail?
If this issue has been triaged, please comment . If you aren't able to handle this issue, consider unassigning yourself and/or adding the label. I am a bot run by vllry.
/unassign
/remove-triage unresolved
To check kernel-modules instead of functions is not limited to in k8s. I build my own kernel which was static from the start (with ipvs support). It was a very tiresome trial-and-error time to find out which functions k8s is checking by it's modules. But those who build the Linux kernel probably can probably accept less "help", so a flag to the apropriate k8s programs may be a solution.
I've checked the code, this is only a warning. Maybe while loading the modules, this function could be reduced to something like "check if this exists exists in the kernel builtin modules and append it into the mod list, otherwise try to modprobe it". I've made a PR to it, let's see if this solves the "problem"
/assign
Those are not options, as compilling the kernel doesn' t imply that one will copy the config file into /boot after recompiling it. As an example, the step ` is optional. So the only way right now I've found to be plausible is to verify the existence of the related but what happens if this path changes in the future?

take a look at to see if this is some way to follow Don't think adding a new flag is a good idea (but I might be wrong either, and TBH I've tried ;) )
rikatz, that looks like a reasonable approach. Although the kernel API isn't standardized, the developers do treat /proc as close to stable as possible, and only change existing files and structures as a last resort (they realize that change here ripple across critial userspace, including stuff like systemd). So I think checking proc for the desired features is the first and most reliable approach and then falling back to those in the order that you've suggested in your write up. The error message can even indicate the name of the /boot/config-* file, to help those building their own kernels who might have forgot to provide a correspondingly-named config.
/reopen The patch was a partial solution, We still need to verify if the module loader will support kernels with all builtin
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Em ter, 14 de jan de 2020 às 07:32, fejta-bot <:


Question: // Optional: The target port on pods selected by this service.  If this // is a string, it will be looked up as a named port in the target <del> // Pod's container ports.  If this is not specified, the default value // is the sames as the Port field (an identity map). </del> <ins> // Pod's container ports.  If this is not specified, the value // of the 'port' field is used (an identity map). // This field is ignored for services with clusterIP=None, and should be // omitted or set equal to the 'port' field. </ins> TargetPort intstr.IntOrString `json:"targetPort"` // The port on each node on which this service is exposed.

In 1.0 and 1.1, headless services that specified a mismatched port/targetPort were tolerated (the targetPort was ignored). tightened validation to reject these services. That causes problems when upgrading from currently-working clusters.
2 candidate to avoid breaking backwards compatibility
oh hell. Yeah. I don't think this is worth a breaking change. We should either roll that back ASAP or change it to nuke any user-provided value and replace it with the value. I think the latter is potentially dangerous and also not worth the risk, so I think we should rollback and just update the docs for targetPort. objections?
Who has time to do this? It doesn't look like an auto-rollback is doable.
Already opened a PR in
Yes, I agree this isn't worth a breaking change, But could we give an warning log that targetPort isn't equal to port and would be ignored?
There's no facility in validation for warning. I don't think spamming the server log in response to API requests is that helpful here
Yes, that sounds reasonable. I just think that targetPort was ignored is a little unfriendly to user.


Question: } func TestEnsureLoadBalancerDeletedDeletesInternalLb(t *testing.T) { <del> t.Parallel() </del> vals := DefaultTestClusterValues() gce, err := fakeGCECloud(vals) require.NoError(t, err)

Since merged on Friday night, my PR () has failed several retests of pull-kubernetes-bazel-test. My PR is unrelated to the load balancer so I suspect that introduced a flake. I see several runs of that test against master that aren't failing, so clearly the test works sometimes (and the original PR passed it several times to before it merged). Perhaps the test harness sometimes runs problem tests in parallel and other times it doesn't. Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: My PR had one good run of pull-kubernetes-bazel-test before merged, but has failed several times since with no code changes to my PR. What you expected to happen: The tests should consistently pass unless related problem code is introduced.
/cc /cc /sig gcp /kind flake
My PR started failing recently too. See https://k8s-
Hey, thanks for identifying the issue. contains tests for - but the race condition was caused when tests with modified a singleton variable in a mock. The short-term fix is removing the calls, but the long-term fix is to not use a singleton. Fix is .


Question: SRCS = consumer consume-cpu/consume-cpu ARCH ?= amd64 TARGET ?= $(CURDIR) <ins> GOARM ?= 7 </ins> GOLANG_VERSION ?= latest SRC_DIR = $(notdir $(shell pwd)) export

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Building the image binaries requires the GOARM variable to be set. Because it is not defined in all Makefiles, cannot the image binaries for the images in question. What you expected to happen: should succeed. How to reproduce it (as minimally and precisely as possible): git clone cd kubernetes/test/images/entrypoint-tester make Anything else we need to know?: Environment: N/A Kubernetes version (use ): N/A Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): N/A Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
/sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or .

/lifecycle stale
/lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten


Question: // makeHostsMount makes the mountpoint for the hosts file that the containers // in a pod are injected with. <del> func makeHostsMount(podDir, podIP, hostName, hostDomainName string) (*kubecontainer.Mount, error) { </del> <ins> func makeHostsMount(podDir, podIP, hostName, hostDomainName string, hostAliases []v1.HostAlias) (*kubecontainer.Mount, error) { </ins> hostsFilePath := path.Join(podDir, "etc-hosts") <del> if err := ensureHostsFile(hostsFilePath, podIP, hostName, hostDomainName); err != nil { </del> <ins> if err := ensureHostsFile(hostsFilePath, podIP, hostName, hostDomainName, hostAliases); err != nil { </ins> return nil, err } return &kubecontainer.Mount{

Is this a request for help? No What keywords did you search in Kubernetes issues before filing this one? Ingress, hosts and /etc/hosts Is this a BUG REPORT or FEATURE REQUEST? (choose one): Feature request There have already been a few issues regarding modifying : , But both have a different use-case. First some information before I explain my use-case. I have a pod which run a web app (nginx container + php-fpm container) and it available over multiple sub domains. I want request to the exposed sub-/domains (ex. ) to go directly to the local webserver instant of going the long way (Cloudflare -Ingress Controller -Node -Container). There is 2 reason behind this: is faster/reduce overhead. of our app code call the exposed domains, and need to hit the same server, as the request oriented from (the code write to some tmp file, and other PHP code need to read that tmp file, when it get a request. Bad design: properly). cc Kubernetes version (use ): Environment: DigitalOcean, CoreOS.
I'm not sure I understand the use case, especially the 2nd. This is the case of a pod talking to itself via the public hostname? Why not just use localhost?
Public hostname == ingress . Yes, without going through the load balancer, some of the PHP also require "sticky" sessions and then it is just easier if the exposed domains just point to localhost. That require extra logic in the app code and when the app code expose multiple sub-/domains, then it is just easier adding the domain to /etc/hosts.
Do you know all the hostnames at pod startup time, or can they possibly be later? I guess I am not against this idea in principle, I just have a nagging suspicion it will blow in all the corner cases. I guess we could look at a community implementation, if someone did one. I don't think that we have time on our side right now to drive it.
I currently have a script like this: , which "extract" the hostnames from the nginx config at container startup (and "patch" the location of the file). It also configure to forward some request internally (we don't expose all domains externally).

-- I'm not sure about your particular use case as the host name extraction logic seems to be quite complex (parsing of the nginx config). The two things I do see that are useful: Ability for a ConfigMap to be written out as a single file into the container instead of injecting it as a directory of files. This would solve a lot of problems where a piece of the container configuration needs to be trivially changed. I'm fairly certain this is already filed, I can't seem to find the issue. Inheritance of host entries from the outer node.
I think the use-case beyond service is if the hosts need to be loopback or needs to be configured per pod. I tried to understand how is controlled by kubelet, and it seems like this feature would not be hard to implement. I'm willing to spend some time on this. in terms of implementation: from my reading, host mount is done by inside , and the content of the hosts file is controlled by the methods that call them all have access to the pod object. it seems like we can add a field to PodSpec (can be done as an Pod-level alpha annotation to start): or if we want something more explicit: that can be read into and , which then mounts as to the Pod.
Let me know if ^ makes sense. If so, I can make a formal proposal in or just a PR, whichever is appropriate.
I don't think this warrants a proposal or feature. I would argue for the more expliciti form of the API, but hostnames has to be plural, and IP has to be the mergeKey. or - any objections? Consider from a CRI point of view?
For me, this solves my problems.
so the struct would be something like where the keys are IP addresses, and values are hostnames. I presume this field would live under PodSpec. is the key we want to surface? ? the content would be: which after some merging would look like: (the go struct represented in yaml) and the output could either look like: or I think the latter is better since it's one line per entry, and seems to follow existing convention in .
Your proposal can definitely work for my use-case, although I would love if I could configure it from a init container, but this could work.

Keep up the good work :)
I wouldn't try to massage it into a map, just: , Kristian Klausen
So you'd be passing in into . That makes sense. This might be a stretch, but what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP.
This is currently handled completely by kubelet, so it wouldn't be an issue for CRI. kubelet doesn't mount if the pod is using the host network, so this would either need to be changed, or validation should reject the pod if this is not allowed. If you need to dynamically change the file after the container starts, I think suggestion of trying to make ConfigMap work would be better.
I think validation to reject the option if the pod is using the host network makes sense. is still a static file though. And if you inject a file that replaces , what are you going to do with existing dynamic hostnames that are by kubelet?
I don't have any concern from CRI perspective, and the proposal look fine to me from the high level. what about different types of IPs that might not be present at the time the configuration is written? like for example the host's IP, pod's IP, or even some service IP. But can I request not stretching the feature to support dynamic changes? Pod can be deleted, and re-created, thus the pod's IP might be changed. Currently we only allow container image being updated in PodSpec. Unless we allow updating the new field, let's call it etcHost for now, in PodSpec, it won't work. Even we allow that field being updated, I think that is overkill.
With regards to the dynamicness, I'm just trying to design the API in such a way that's flexible. The field in PodSpec won't be dynamic. I imagine something like this: (The idea is borrowed from ) validation ensures that is only present if type is . then IP is gathered at Pod creation time. In kubelet's :
Why do you need pod IP? Why not use 127.0.0.1 ?

, Ricky Pai <:
The use-case that i'm more interested in is support for nodeIP, but I'm using podIP as an example because it's easier to illustrate with current code. But yes I agree with you that pod IP might not make sense since you have access to localhost already. I can imagine nodeIP being used to talk to daemons that are shared across pods. is a proper fix for the use-case, but this could be one way to address it.
sorry for the scope-creep. I'm just afraid using IP as a string field makes it hard to add support for other IP sources should that ever be an option. but it's probably premature right now. for now, I'll just proceed with:
Given that we have a way to get node IP through downward API, why do we need that here? On Apr 13, 2017 6:09 PM, "Ricky Pai" <:
Ideally, if the applications running inside the containers have flexible ways to configure dependent hostnames like through environment variables, then downward API would be sufficient. We can probably extend the argument that the whole support for is moot if applications supported flexible injection of dependent hostnames. The problem is many applications don't provide this level of flexibility. Hostnames are often hard-coded inside config files and the client libraries. Outside of Kubernetes, I might use a script to manage and expect nothing else to change the file; while it is possible that I use an init container to do the same modification to , since the mounting of happens whenever a container is restarted, I risk the file being changed and my application not being able to connect to its dependent services. This creates much headache migrating applications from a more lenient environment to Kubernetes, which is what we've been facing. I think support is the right direction and is why I'm willing to commit time to make it happen.
We have another issue open for "node-local Services" which sounds like what you want. I am not a big fan of the expansion of host mappings like this. I'd rather go all the way and give nodes a static link-local address or else solve node-local services. , Ricky Pai <:
Thanks for all the feedback I'm well aware of node-local services, but it seems like progress on the proposal is stalled.

I believe node-local services would be the proper way to address the use case I described, especially with the "locality" concept you described in that can be expanded beyond the node. However, in lieu of node-local services, with references to the node IP might be a quick way to address the use case. Other tickets describing the need for node-level sidecars also proposed either using node-local services or : I also understand this complicates the interface a little bit more, but it would really benefit cluster operators. If you still think this is inappropriate, then I can proceed with this interface next week:
I don't have a strong opinion on this topic but it seems that it should be addressed as part of what is the status of the proposal?
I am not familiar with the node-local services and its progress, but if this can satisfy the use cases, I'd suggest that not adding new fields to the pod API for a quick workaround.
I feel the original proposal got digressed by my zealous talk of node IP support. Allow me to clarify: serves several use cases: external hostname with a specific IP. i.e. if I want to say mock out an external hostname like a hostname with localhost. i.e. if I want to override to point to we allow dynamic injection of nodeIPs, it can be an easy way for pods to talk to nodes in absence of node-local services. only the third use-case is relevant to node-local services. I think it's still useful to support adding entries to even with static IPs
I understand the third case, but I'd like to consider other options to solve that, first. I think the rest is uncontentious. , Ricky Pai <:
Thanks. I'll get the changes up this week. There'll likely be three changes: tests for existing hosts file content (there's no tests currently) change for on to use in Is there any possibility to cherry-pick those back in 1.6? Or would this have to wait for 1.7?
AFAIK, we don't cherrypick features.
has been created for the API change
just to clarify, are you in favor of adding HostMappings to the API?
I am neutral on it. It's come up often enough that i believe the problem is real. The API doesn't seem harmful to me, what do you think?

You're the guardian of CRI... , Yu-Ju Hong <:
Also the name is terrible - help? HostAliases? DNSAliases? , Tim Hockin <: solve that, first. I think the rest is uncontentious. , just to clarify, are you in favor of adding HostMappings to the API? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub , or mute the thread
I'm up for . might accidentally imply that it's actually interfacing with kube-dns.
I am neutral but just wanted to make sure there are no obvious, alternative ways to achieve this. BTW, the proposed solution changes the k8s API, but not CRI :) On the other hand, if we support this on the k8s API level, we should discourage users from modifying this file directly. For example, reported a bug where users modify in the init containers, and later found that kubelet overwrites it.
We should definitely discourage modifying that, but we should perhaps respect it if a user provided a volume mount. Thoughts on name? , Yu-Ju Hong <:
I am terrible at naming... I agree that having DNS in the name can be misleading. sounds ok. might be more direct?
I'd go for brevity when possible. HostAliases ? On Apr 27, 2017 5:27 PM, "Yu-Ju Hong" <:
I'm updating the PR to use . thanks for the suggestion
Just posted the last PR to enable this feature:
could you add a node e2e test for this feature?
Do we need e2e? Every e2e is a liability, and slows down test runs - e2e is needed if we can't cover in unit or integration. , Yu-Ju Hong <:
Unit tests are great to ensure the individual functions work correctly, but so far it's not sufficient to prevent regression in our codebase from my experiences. Since this is a new addition/feature to the API, I'd prefer having a corresponding node e2e test. Integration test can be sufficient, but I've not had any pleasant experience debugging integration tests. Maybe that has changed already?
no, integrtaion tests are under-serviced tech-wise.

, Yu-Ju Hong <:
does a simple node e2e test sound good to you?
SGTM , Yu-Ju Hong <:
I saw that the HostAlias's comment: I'm wondering why it couldn't work to hostNetwork pods? And how does a hostNework pod initialize its /etc/hosts file?
Are we still missing a test for this? I'd hate to have to kick it out of the 1.7 release for lack of tests..
yes we are still missing a test. what's the deadline for 1.7 release? sorry i was on PTO and dropped the ball
hostNetwork pods uses the host's file, while only non-hostNetwork pods have managed by kubelet. because kubelet does not manage the file when using hostNetwork, kubelet can't write additional entries from hostAliases.
Just chatted on slack, for containers using the host network, docker actually makes a copy of "/etc/hosts" on the host for each of them. I think kubelet can potentially support HostAliases for hostnetwork pods by copying and mounting the file. Not sure if there are be other concerns though.
if you make the change to make kublet copy and manage for hostNetwork pods, I can make the change for HostAliases. one concern: currently, the logic includes , including from and . Should this be allowed? if not, there should probably be validation disallowing those field when hostNetwork is used. there currently isn't any unless I read the code wrong.
node e2e test:
THANKS! , Ricky Pai <:
That would be a great feature! Currently we have pods using hostNetwork and using init containers to restore some host mappings -- basically . With HostAliases feature it would be fantastic fit!
We are using and for hostNetwork pods.
This is something I don't understand in the code: It seems that it is still mounting "/etc/hosts" for hostnetwork pods if . Does that mean kubelet is still managing it?
One of our current assumptions is that kubelet will mount a shared /etc/hosts (which is not node's) for all containers within a pod in hostNetwork mode. Which seems to be the codepath in the code linked above. Is this holding true?
I checked the git history and that was just an artifact caused by switching from internal to v1 API objects.

The original code is: In the internal API, is a field in , so you can see why had to be non-nil. In the v1 API, is a field in the pod spec: . When switching all code to use the versioned (v1) objects, this got lost in translation. I believe the apiserver sets the default pod SecurityContext, so it should never be nil. I'll send a PR to remove the redundant check.
Thanks for looking into that. Currently even with hostNetwork pods, it will have mounts for . Checking within the pod: If the check is removed, what would happen on the shared mount of /etc/hosts ? My concern and question is about the guarantee here. Is there still a copy of which is shared and mounted for all containers within a pod?
Removing the check itself won't affect anything since securitycontext should always be non-nil. That mount for hostnetowrk pods was by docker (not by kubelet) so it should be one per container, and not shared by containers within a pod. I didn't try it myself, but it should be easy to verify.
Tried and verified that was shared between containers within a pod for k8s 1.6.4 .
Also verified that modifying for a hostNetwork pod did not change the file on the node.
Then docker did the right thing and use the same file for containers sharing the same network namespace :-)
That makes sense.
Will HostAlias support for HostNetwork pod happen in 1.7 ?
I'm trying to get in so it can make the 1.7 release deadline. help reviewing the PR would be great.
in a multi-container hostNetwork pod, what happens when you change the file, then restart one of the containers? in non-hostNetwork pod, it would revert the changes to
anyways, i'm also in favor of kubelet managing the file even for hostNetwork pods. that way, we can use features like hostAliases, and also then we don't depend on the container runtime to dictate the behavior of the hosts file. it's not clear whether running rktnetes with hostNetwork pod would yield the same result
Is this a dup of ? How are they different?
HostAlias is useful to provide local overrides for a particular Pod, but it seems like would want the alternative DNS name of a service accessible for all Pods.

re: documentation, i'm thinking either adding notes about HostAliases in or as a separate page under "Services, Load Balancing, and Networking"
Is there any issue created for supporting HostAlias in hostnetwork mode?
I can write up what I think will be needed to support HostAlias in hostNetwork pods
This is now in 1.7:
filed issues to follow up with HostNetwork support:
Thank you for this feature. Our use case is different. On an nginx/php-fpm set up where our local dev is via docker-compose yet we're deploying into kubernetes, we can use the exact same nginx config on both environments. We're currently working around this via HOSTALIASES (alias or or whatever the php container is into , which satisfies both deployment types), but this is hacky and doesn't work in Alpine.
the discrepancy between dev/prod would probably cause problems beyond just difference in networking setup. have you considered using to have parity between the two environments?
We have, and found it way too cumbersome for local development. It's way heavier in resources as well - obviously there's a lot more than just docker needed for minikube to run, and most of our developers use Mac, where all this is virtualised. In contrast, docker-compose is a lot easier, more concise, and we have yet to find any environment-y issues caused by difference between these two, except for this one issue we fortunately have a workaround for (HOSTALIASES). We use the exact same base container images for development as well as builds. From the point of view of the services themselves, there's no difference at all being run on one environment or the other - they talk to each other via HTTP or queues.
that's unfortunate. I guess minikube does have overhead since you have to run it inside a VM on your mac machines. But since all components are written in go and cross-compiled in darwin, I wonder if you could setup your own etcd (homebrew has this), then your own kubelet. Docker you can run Docker on Mac, then just have kubelet talk to the local docker socket. This sounds like it would work with lower resource overhead than minikube since you don't to run in a VM. Either way, I'm glad HostAlias is able to address your problem.

I just wish you didn't have to deal with this discrepancy in the first place.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. Prevent issues from auto-closing with an comment. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/close


Question: klog.V(4).Infof("====Applying Policy for %s====", svcName) // Create Remote endpoints for every endpoint, corresponding to the service containsPublicIP := false <ins> containsNodeIP := false </ins> for _, ep := range proxier.endpointsMap[svcName] { var newHnsEndpoint *endpointsInfo

What happened: Executing inside a Windows Pod, but get the following msg:  What you expected to happen: Executing inside a Windows Pod, could view: How to reproduce it (as minimally and precisely as possible): Flannel mode to create a Cluster with one control plane (via RKE) and one Windows Node (via ) into the Windows Pod is fine is fine , but get timeout error Anything else we need to know?: Windows workload yaml:Nodes info:  Content of from :More researching:Deploy an Nginx workload with host port exposed, and then create an Endpoint and Service for it, finally access the vip from Windows inside:   Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ): Linux: Windows: Kernel (e.g. ): Linux: Windows: Install tools: Control plane created via RKE, Windows node created by Kuberentes guide Network plugin and version (if this is a network-related bug): Flannel VxLan
/sig windows
More information If we edit the default kubernetes endpoint to point to the node’s public ip, then works
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
can you provide the Kubernetes version for the windows node? I see platform as linux in your post
Hi , it happens in 1.14, 1.15 and 1.16. You sent us a commit to fix it before. It works well. Hope we can get it fixed in upstream as well.
In which version was it fixed?

The k8s version I use is: windows Server core version: The problem still appears
The fix for the issue above is included in 1.18.1 to help further
But it doesn't work in 1.18.1
Verified in K8s 1.18. It only works in some particular Windows versions. I guess it is a Windows (OS) issue. works. doesn't work. If you are using AWS, please use instead of the current default one . CC
I do n’t think it should be a system problem, I am using Microsoft Windows Server version 1909 [OS build .778]
: we have identified a potential issue in this area last week. In order to make sure we are talking about the same issue, would it be possible to collect some logs using the instructions below, please? an elevated powershell Window on the host and run: wget -o collectlogs.ps1 the same command line, run .collectlogs.ps1 share the resulting logs folder
Here is a link to the issue I mentioned in my previous comment: Though the issue description is different, the end result is the same: is not accessible from within the POD
any news about that? I'm also running into this issue with newer Windows Versions and can't stay on build .1039
: Two fixes were merged to the master branch a couple of weeks ago. kubernetes/kubernetes kubernetes/kubernetes AFAICT there is no kubernetes releases yet that include these fixes, but I am expecting the next v1.19.x to include these.
and FYI.


Question: allPods := getPodsByLabels(c, ns, map[string]string{"name": labelName}) scheduledPods, notScheduledPods := e2epod.GetPodsScheduled(masterNodes, allPods) <del> printed := false printOnce := func(msg string) string { if !printed { printed = true return msg } return "" } framework.ExpectEqual(len(notScheduledPods), expectedNotScheduled, printOnce(fmt.Sprintf("Not scheduled Pods: %#v", notScheduledPods))) framework.ExpectEqual(len(scheduledPods), expectedScheduled, printOnce(fmt.Sprintf("Scheduled Pods: %#v", scheduledPods))) </del> <ins> framework.ExpectEqual(len(notScheduledPods), expectedNotScheduled, fmt.Sprintf("Not scheduled Pods: %#v", notScheduledPods)) framework.ExpectEqual(len(scheduledPods), expectedScheduled, fmt.Sprintf("Scheduled Pods: %#v", scheduledPods)) </ins> } func getPodsByLabels(c clientset.Interface, ns string, labelsMap map[string]string) *v1.PodList {

What happened: When an issue happens at the following check in the conformance test "validates resource limits of pods that are allowed to run": the debugging message ("Scheduled Pods: %#v") doesn't appear in the log at all like the following: ` and that makes debugging difficult actually. What you expected to happen: The debugging message should appear in the log if an issue happens. Environment: Kubernetes version (use ): v1.15.1 Cloud provider or hardware configuration: OpenStack OS (e.g: ): Ubuntu 18.04.1 LTS Kernel (e.g. ): 4.15.0-55-generic Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
/sig testing /sig scheduling


Question: }, { kubernetesVersion: "1.10.0", <del> expectedVersion:   version.MustParseSemantic("3.1.11"), </del> <ins> expectedVersion:   version.MustParseSemantic("3.1.10"), </ins> expectedError:     nil, }, {

uses non existing etcd version for k8s 1.10.0. /kind bug /sig cluster-lifecycle


Question: Containers: []v1.Container{ { Image: busyboxImage, <del> Name:  podName, </del> <ins> SecurityContext: &v1.SecurityContext{ SELinuxOptions: &v1.SELinuxOptions{ Type: "container_logreader_t", }, }, Name: podName, </ins> // If we find expected log file and contains right content, exit 0 // else, keep checking until test timeout Command: []string{"sh", "-c", "while true; do if [ -e " + expectedLogPath + " ] && grep -q " + log + " " + expectedLogPath + "; then exit 0; fi; sleep 1; done"},

<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --ci-crio-cgroupv1-node-e2e-conformance ContainerLogPath It was failing last week, then I disabled the SELinux while we investigated the issue Once this PR merges, I will enable the SELinux again for CRIO jobs. Once the SELinux is turned on and all the required components in the OS (e.g. crio, runc) are compiled with SELinux support we observed following errors in audit logs, Which points towards incorrect SELinux labeling.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node


Question: if found { data, err := ioutil.ReadFile(path) if err != nil { <del> glog.Errorf("Error on reading termination-log %s: %v", path, err) </del> <ins> result.status.State.Terminated.Message = fmt.Sprintf("Error on reading termination-log %s: %v", path, err) </ins> } else { result.status.State.Terminated.Message = string(data) }

Error on reading termination-log /var/lib/kubelet/pods/-f986-11e4-9c82-/containers/skydns/933ddf81dbaf586e425240432429aa57bec939383e19bfb25d35d478597203f7: open /var/lib/kubelet/pods/-f986-11e4-9c82-/containers/skydns/933ddf81dbaf586e425240432429aa57bec939383e19bfb25d35d478597203f7: no such file or directory
I think there is a regression lately breaking termination-log.
might be related to this issue.
Was this fixed by ?
No, I still observed such error logs once a while, but so far those errors are red herring, without any breakage. Haven't get down to look into the detail yet.
If a pod gets deleted from the apiserver, kubelet (more specifically, the SyncPods function) would stop the container, and remove the orphaned volumes and pod directories. In the meantime, the per-pod workers (goroutines) may not have terminated yet. Therefore, the workers could be inspecting the containers to generate the status, causing the errors. Note that this does not happen if containers terminated by themselves while the pod is not yet deleted. The error messages are harmless, since the pods have been deleted and no one would examine the termination logs anyway. However, the fact that is full of such errors is misleading and confusing to users. I think this would be solved by graceful termination...