Question: // Only the addresses of the current family are returned. // IPv6 link-local and loopback addresses are excluded. GetLocalAddresses(dev string) (sets.Set[string], error) <ins> // GetAllLocalAddressesExcept return all local addresses on the node, except from the passed dev. // This is not the same as to take the diff between GetAllLocalAddresses and GetLocalAddresses // since an address can be assigned to many interfaces. This problem raised // https://github.com/kubernetes/kubernetes/issues/114815 GetAllLocalAddressesExcept(dev string) (sets.Set[string], error) </ins> }

Hey, after upgrading kubernetes from 1.22 to 1.23, I'm experiencing strange service provisioning behavior using nodePort. I'm using kube-proxy (ipvs) + calico + metallb. If external IP is used by another service type (Loadbalancer - allocated by metallb), then the node owning this IP doesn't listen to NodePort on this interface. Unfortunately, I can't find any changes that could be causing this, any ideas? Kubernetes 1.23 Missing externalip1 here: Using k8s 1.22 service with nodeport caused listening on every interface, despite using external IP by another service. Kubernetes 1.22 Upgrade k8s from 1.22.15 to 1.23.15 No response <details</details<detailson-premise instances </details<details</details<detailskubeadm </details<detailsdocker </details<detailscalico v3.24.5 metallb v0.13.7 ipset v7.5, protocol version: 7 ipvsadm v1.31 2019/12/24 (compiled with popt and IPVS v1.2.1) #ipvs config: </details
/sig network
/assign
From : This is not a valid setup. A virtual IP (VIP), like the loadBalancerIP, must not be assigned to a physical interface. The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without_ assigning the VIP to any interface.
When you send traffic to a nodePort, the destination address should be a node address, not a loadBalancerIP.
There is however a change in the implementation (I will try to find the PR) in the way the nodePort addresses are computed. This is how i should be; On all clusterIP, loadBalancerIP and externalIPs are assigned. So the current implementation is correct. What I suspect that pre v1.23 K8s did this wrong by adding all addresses on interfaces except kube-ipvs0 (including loopback which would not work).

The reason to assign a VIP to a physical interface would be to attract traffic with L2 mechanisms, ARP for IPv4 and neighbor discovery for IPv6, but this way is flawed for several reasons. That's why you use metallb in L2 mode instead. Metallb in L2 mode answers L2 requests without assigning the VIP to any interface. These loadBalancerIP addresses are just the external IP of my nodes, which I use in metallb as an available pool for service(loadbalancer). Provider automatically assigns this address to the physical interface as a lease, I don't really have the option of a different setup at the moment. Ye, I know. In my case the public IP address pool for node and loadBalancerIP addresses are the same
So, the loadBalancerIPs assigned by metallb is actually the real addresses of your K8s nodes?
Yes, these are public IP addresses (each node has its own), which I use as an entry point for some services(loadbalancer) but I also have an external loadbalancer that targets nodePort. To eliminate SPoF, it directs traffic to the service (NodePort), which has been listening on all nodes so far, regardless of IP assignment to another service
Is this setup supported? Thanks for explaining. I understand your setup now, . I am unsure if it's supposed to work, so I have to ask. Have you tested proxy-mode=iptables? It may work, but I wouldn't be suprised if it doesn't.
I will try to test it with iptables, but I use ipvs because of specific services, so I can use the load balancing algorithm - least connection... this method won't solve my problem :(
Found the PR
Why do you need nodePorts services at all? Can't you use only loadBalancer? NodePort is mainly intended to be used by and external load-balancer, i.e. like in AWS, but when you manage your own cluster you usually don't have an external loadBalancer so use only loadBalancerIPs makes sense. We do that, and even .
Scratch that! In your case where the node addresses are the external addresses you should use NodePort only. If you use your node addresses as loadBalancerIP you will hit this PR when you upgrade to later K8s versions.

It inserts a blocking rule in the INPUT chain for loadBalancerIPs. This will block input to your node if loadBalancerIP==nodeIP.
I understand, thanks for the info. In my case it's a breaking change and I will have to migrate some services to NodePort before kubernetes upgrade on prod environment.
with proxy-mode=iptables, everything started working, node is listening on nodePort and I can get to loadBalancerIP, I cleared the rules on each node, just to be sure However this is not a solution for me, I need the lc algorithm from ipvs
Port used in the LoadBakancer service example is within the default NodePort range. I think it can be converted to a NodePort service without affecting external users. Or is just an example? Or are there other reasons for type:LoadBalancer?
/area ipvs
This is just an example, I won't have much trouble with the migration. At most, I will divide the available lease of IP addresses into smaller pools, for NodePort / Loadblancer services. I think I already know everything and there should be no more surprises. Thanks for the information and your time!
It is a little "unusal" to use the host IP as the LB IP, but it's unfortunate that we broke something that used to work. Lars, is there any path back to "working" that doesn't lose the protections in that linked PR? I could easily see iptables also breaking this..
To not exclude loadBalancerIPs (and ecternalIPs) when they are on a physical interface is not hard, but messes up the nice set-operation. The access protection in It should basically not be there if the any loadBalancerIPs (and ecternalIPs) is also on aphysical interface. Can be done, but may be messy.
/triage accepted Even though this issue is accepted, it's still under investigation. I will check the code if the old bevaivor can be restored in a fairly simple way. But there is a problem with back-port. If I make a PR now it will be in v1.26.x or v1.27 but the function was broken in v1.23 so a fix should really be back-ported all the way back.

I would prefer to document that a real address on an interface on the node can't be used as a virtual address, like loadBalancerIP or externalIP.
Just wanted to chime in and say that I've also encountered this issue... very grateful to find the discussion and looking forward to a solution. Thank you all! Also just to share, I've tried the following 3 workarounds and they all work in my case: Switching to iptables mode. It works I think because the iptables NAT rule refers to the hostname as the destination, which gets resolved to the public node IP. Disable IPv6 on my OS. This works too as it makes the code fall back to the single stack mode, which grabs the node IP also from looking up the hostname. Explicitly configure the field in the kube proxy config, that overrides the default get local addresses.
I backed down to K8s v1.22.4 and I can't get it to work unless you have the master outside the cluster (i.e. not a node), or have a setup where the external-ip you set as loadBalancerIP is not on the main K8s network, something like; <img src="" width="65%" /K8s is using the "backend" network, the "external" ips are on the "frontend" network. The reason is when you assign a loadBalancerIP, that address will be assigned to the interface, even before v1.23. So, on an in-cluster master node you will see something like; The address is the loadBalancerIP and is also the node-address of another node in the cluster, "vm-002". When the master receives "normal" packets from "vm-002" they will have src=192.168.1.2 but since that address is assigned to a local interface on the master the packets will be discarded as or spoofed packets. The result is that "vm-002" (e.g. on that node) loses contact with the master;
I discovered this when testing PR I think I can restore the pre v1.23 behavior now, but I suffer from the problem described above.
I don't think your problem is precisely the same. To disable ipv6 or set nodePortAddresses can't really be used as work-arounds for this issue.

Thank you, You're right about IPv6 - disabling it didn't really help. I forgot to remove the I put in while testing with disabling IPv6. It was the that makes the difference. That said, it really appears to work if we explicitly put in the . Maybe I'm still missing something in the code, but if I read it correctly, here is where we take out the load balancer IP: It is called from the folllowing, but only if contains any zero CIDRs. If a node port address is explicitly provided and it is NOT a zero CIDR, then will not be called. We're planning to roll out this workaround before upgrading to a future Kubernetes release with your fix. We would much appreciate it if you could shed more light why it wouldn't work. We didn't want to change to iptables mode if this second workaround really works. Thank you so much!
You are right. To explicitly set in the kube-proxy config works as a work-around. Still the loadBalancerIP (which is also an address on a node) can't be used by the main k8s network, or the master must not be a node in the cluster as described in And... This will work up to and including K8s v1.25.x, but NOT v1.26.0! I haven't checked yet but I think it's because of
Since the loadBalancerIP that is owned by a node is assigned to on other nodes you must stop those other nodes from responding to ARP. For example with:
From my perspective, this is not a big , I have the whole architecture implemented in separate networks. I think it's worth adding this information somewhere in the documentation.
Thank you so much for the response, epesically the info about v1.26. We'll check it out for sure and will report back. Our node port use case is for single-node deployments only, so luckily there are no other nodes.


Question: numActionsBefore := len(client.Actions()) + 1 standardSyncService(t, esController, ns, serviceName, "false") <del> if len(client.Actions()) != numActionsBefore+5 { t.Errorf("Expected 5 more actions, got %d", len(client.Actions())-numActionsBefore) </del> <ins> if len(client.Actions()) != numActionsBefore+2 { t.Errorf("Expected 2 more actions, got %d", len(client.Actions())-numActionsBefore) </ins> } <del> // endpointslice should have LabelsManagedBy set as part of update. </del> <ins> // only 2 slices should match, 2 should be deleted, 1 should be updated as a placeholder </ins> expectAction(t, client.Actions(), numActionsBefore, "update", "endpointslices") <del> // service should have managedBySetupAnnotation set as part of update. expectAction(t, client.Actions(), numActionsBefore+1, "update", "services") // only 3 slices should match, 2 of those should be deleted, 1 should be updated as a placeholder expectAction(t, client.Actions(), numActionsBefore+2, "update", "endpointslices") expectAction(t, client.Actions(), numActionsBefore+3, "delete", "endpointslices") expectAction(t, client.Actions(), numActionsBefore+4, "delete", "endpointslices") </del> <ins> expectAction(t, client.Actions(), numActionsBefore+1, "delete", "endpointslices") </ins> } // Ensure SyncService handles a variety of protocols and IPs appropriately.

What happened: kube-controller-manager panic for What you expected to happen: no panic. How to reproduce it (as minimally and precisely as possible): No idea. I guess it's hard to reproduce. Anything else we need to know?: Found from here: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /assign cc Should we do something for v1.17? (v1.17 code freeze now, but still have the opportunity to fix) can be removed safely in v1.18, right?
/sig scheduling
Working on this in /sig network /priority critical-urgent /assign
Sorry, I didn't realize it. I should have a search before firing an issue. I recommend adding this issue to your PR. e.g. `
Updated, thanks!
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
This seems to be writing to a cached (should be const) service. I think we need to DeepCopy() it before we mutate the Go type. does that sound right? Is this still our best state of the art?
yes, copy before change
we have a tool to catch this (the cache mutation detector), but it got disabled in our test runs two years ago (xref )
yes great catch - I was always convinced this is enabled...


Question: RegisterProvider("skeleton", func() (ProviderInterface, error) { return NullProvider{}, nil }) <del> // The empty string also works, but triggers a warning. RegisterProvider("", func() (ProviderInterface, error) { Logf("The --provider flag is not set.  Treating as a conformance test.  Some tests may not be run.") return NullProvider{}, nil }) </del> <ins> // The empty string used to be accepted in the past, but is not // a valid value anymore. </ins> } // SetupProviderConfig validates the chosen provider and creates

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind cleanup What happened: In PR we tightened the --provider parameter checking of the binary such that it only accepted known providers. This turned out to be problematic () because people had started to depend on the previous behavior, which was to accept anything and merely warn about it. For the sake of giving people time to adapt, PR reverted to the older behavior. What you expected to happen: In 1.14 we want to allow only known providers. While at it, the error handling for unknown providers should be enhanced (print proper error message instead of calling Failf, which panics). /assign /sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
kubeadm was one of the tools which got broken by the tightening of the parameter checking. Does kubeadm still rely on the acceptance of unknown provider parameters or can the current warning be turned into an error now? /remove-lifecycle stale
yes the kubernetes-anywhere deployer is still bound to provider=kubernetes-anywhere (unknown) in test infra jobs. we went back and forth on this topic, but if you wish to replace this support for unknown deployers i can switch those jobs to . LMK.
given one release has passed with the deprecated message on the CLI, it's probably safe to remove this functionality now. :)
"Lubomir I. Ivanov" <: Yes, please do. Once that is complete, I'll turn the warning into an error.
done:


Question: apiserverCurrentInqueueRequests = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ <del> Namespace: namespace, Subsystem: subsystem, Name:      "current_inqueue_requests", Help:      "Number of requests currently pending in queues of the API Priority and Fairness system", </del> <ins> Namespace:      namespace, Subsystem:      subsystem, Name:           "current_inqueue_requests", Help:           "Number of requests currently pending in queues of the API Priority and Fairness system", StabilityLevel: compbasemetrics.ALPHA, </ins> }, []string{priorityLevel, flowSchema}, ) apiserverRequestQueueLength = compbasemetrics.NewHistogramVec( &compbasemetrics.HistogramOpts{ <del> Namespace: namespace, Subsystem: subsystem, Name:      "request_queue_length_after_enqueue", Help:      "Length of queue in the API Priority and Fairness system, as seen by each request after it is enqueued", Buckets:   queueLengthBuckets, </del> <ins> Namespace:      namespace, Subsystem:      subsystem, Name:           "request_queue_length_after_enqueue", Help:           "Length of queue in the API Priority and Fairness system, as seen by each request after it is enqueued", Buckets:        queueLengthBuckets, StabilityLevel: compbasemetrics.ALPHA, </ins> }, []string{priorityLevel, flowSchema}, ) apiserverRequestConcurrencyLimit = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ <del> Namespace: namespace, Subsystem: subsystem, Name:      "request_concurrency_limit", Help:      "Shared concurrency limit in the API Priority and Fairness system", </del> <ins> Namespace:      namespace, Subsystem:      subsystem, Name:           "request_concurrency_limit", Help:           "Shared concurrency limit in the API Priority and Fairness system", StabilityLevel: compbasemetrics.ALPHA, </ins> }, []string{priorityLevel}, ) apiserverCurrentExecutingRequests = compbasemetrics.NewGaugeVec( &compbasemetrics.GaugeOpts{ <del> Namespace: namespace, Subsystem: subsystem, Name:      "current_executing_requests", Help:      "Number of requests currently executing in the API Priority and Fairness system", </del> <ins> Namespace:      namespace, Subsystem:      subsystem, Name:           "current_executing_requests", Help:           "Number of requests currently executing in the API Priority and Fairness system", StabilityLevel: compbasemetrics.ALPHA, </ins> }, []string{priorityLevel, flowSchema}, ) apiserverRequestWaitingSeconds = compbasemetrics.NewHistogramVec( &compbasemetrics.HistogramOpts{ <del> Namespace: namespace, Subsystem: subsystem, Name:      "request_wait_duration_seconds", Help:      "Length of time a request spent waiting in its queue", Buckets:   requestDurationSecondsBuckets, </del> <ins> Namespace:      namespace, Subsystem:      subsystem, Name:           "request_wait_duration_seconds", Help:           "Length of time a request spent waiting in its queue", Buckets:        requestDurationSecondsBuckets, StabilityLevel: compbasemetrics.ALPHA, </ins> }, []string{priorityLevel, flowSchema, "execute"}, ) apiserverRequestExecutionSeconds = compbasemetrics.NewHistogramVec( &compbasemetrics.HistogramOpts{ <del> Namespace: namespace, Subsystem: subsystem, Name:      "request_execution_seconds", Help:      "Duration of request execution in the API Priority and Fairness system", Buckets:   requestDurationSecondsBuckets, </del> <ins> Namespace:      namespace, Subsystem:      subsystem, Name:           "request_execution_seconds", Help:           "Duration of request execution in the API Priority and Fairness system", Buckets:        requestDurationSecondsBuckets, StabilityLevel: compbasemetrics.ALPHA, </ins> }, []string{priorityLevel, flowSchema}, )

<!-- Please only use this template for submitting enhancement requests --What would you like to be : API priority and fairness metrics have labels such as "priorityLevel" and "flowSchema". These should be "prioritylevel" and "flowschema" respectively. These metrics have been to an exception list: Why is this needed: snake_case is consistent with other metrics. /sig api-machinery /kind cleanup /cc Do we want to do this before 1.20 beta?
yes, lets do that.
/triage accepted Yes, we do not want to go to beta with known wrongness in the metrics.


Question: iptables                    utiliptables.Interface haveReceivedServiceUpdate   bool // true once we've seen an OnServiceUpdate event haveReceivedEndpointsUpdate bool // true once we've seen an OnEndpointsUpdate event <ins> MasqueradeAll               bool </ins> } // Proxier implements ProxyProvider

Related to . As a safety mesure, it could be very useful to actually include a flag to proxy to MASQUERADE everything (acting very close to a userspace proxy). I think it's not very hard to do and a very good way to diagnose or even fallback in case of problem (I'm thinking about vxlan cases). Something like "--snat-clients" or "--snat-pods" could be appropriate.
/cc
Looking at the current code: It looks like we SNAT almost everything already now? Would adding a set-mark rule (behind the flag of course) at line 548 be sufficient? Deprecated Public IPs, Loadbalancer IPs, and NodePorts all appear to be marked for SNAT already (also endpoint loopback traffic).
I think that is the flag, yeah. , Benjamin Elder wrote:
Ok, I'll push out a PR for that in a moment. I should be able to take care of that and the cleanup flag (PR out but needs some more work) today. On Aug 20, 2015 2:12 PM, "Tim Hockin" wrote:


Question: import ( "fmt" <del> "io/ioutil" "net/http" </del> "time" "github.com/GoogleCloudPlatform/kubernetes/pkg/api"

Some of the e2e tests specify HostPort in their pod configurations. This can cause issues running those tests if the port is already in use.
We've been removing these over the last week and a half and I think that may be the only one left remaining.
We need to decide if HostPort is a supported use case for Kubernetes -- and if it's supported, then it needs e2e test coverage to ensure it remains functional. Note that supported and not-preferred are orthogonal. My understanding is that it's supported-but-not-preferred and that there are reasonable use cases for using it.


Question: // Manager interface provides methods for Kubelet to manage pod cpus. type Manager interface { // Start is called during Kubelet initialization. <del> Start(activePods ActivePodsFunc, podStatusProvider status.PodStatusProvider, containerRuntime runtimeService) </del> <ins> Start(activePods ActivePodsFunc, sourcesReady config.SourcesReady, podStatusProvider status.PodStatusProvider, containerRuntime runtimeService) </ins> // AddContainer is called between container create and container start // so that initial CPU affinity settings can be written through to the

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: PR , it will remove "inactive containers" while reconciling. But the cpumanager started very beginning, it will get empty activePods at it first run. This make state file no sense. What you expected to happen: We should wait a moment or notification from podmanager before starting reconcile. Some log may be before remove "inactive containers". How to reproduce it (as minimally and precisely as possible): restart kubelet Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
In reconcileState : You mean the cpu assignments would be wiped for the first reconciliation ?
Yes, you can test it by restarting kubelet.
Do you see any of the following log:
I see this
/assign


Question: }.AsSelector().String() eventListOptions = metav1.ListOptions{FieldSelector: selector} By("Create the test log file") <del> tmpLogFile = filepath.Join("/tmp", name, path.Base(logFile)) os.MkdirAll(path.Dir(tmpLogFile), 755) _, err = os.Create(tmpLogFile) </del> Expect(err).NotTo(HaveOccurred()) By("Create config map for the node problem detector") _, err = c.Core().ConfigMaps(ns).Create(&v1.ConfigMap{

Leads to this avc: In general, HostPath volumes in tests are bad news for selinux.


Question: } func (o *CopyOptions) Run(args []string) error { <ins> if len(args) < 2 { return fmt.Errorf("source and destination are required") } </ins> srcSpec, err := extractFileSpec(args[0]) if err != nil { return err

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: What you expected to happen: I think should just show the help or something? How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): ubuntu 16.04 amd64 Kernel (e.g. ): Install tools: Others:
/sig cli
It has fixed in higher version:
You have 1.9.7, I was testing . There's a regression, it seems. Anyway, it's just a minor annoyance, it just looks ugly, that's all.
This is indeed a bug. /assign
with some arguments crash all the way from 1.9.0 to 1.10.5 (1.8.14 is ok): Command: (no trailing slash): 0: 10.5:
Off-topic, but kubectl 1.10.5 release binary also looks funky with in the path, while up to 1.9.3 it's clean . From 1.9.4 onwards it has changed to ?
Since this has fix candidate going in already, opened a separate issue for my variant:


Question: "io" "net" "path" <ins> "sort" </ins> "strconv" "strings" "sync"

Is this a BUG REPORT or FEATURE REQUEST?: /kind feature /sig aws Anything else we need to know?: AWS NLBs support static IP addresses by associating an EIP per AZ. An already allocated EIP can be supplied with the . A user can then supply EIP allocation IDs in the service definition of kubernetes. By that the service will be available via those static IPs.
eip's for white listing are really important in the enterprise, we need this feature now, as a result we have to manually create nlb's, would really like to see this
This is also important for hosting providers, who might have huge numbers of apex domain DNS records pointed at EIPs. Associating an already-allocated EIP with a new Kubernetes-generated NLB would be huge!
This feature would save us weeks of work reimplementing the cloud controller just so we can preserve a special ELB that has static IP addresses... I really wish it was available
I'd like to contribute time to working on this. Is anyone else actively working on adding support?
That's awesome news. Thanks for working on this. Do you need assistance in testing this in an AWS environment? I might help out with that ...
Sure, the PR has been approved for testing so the binaries are available. I'm moving the code into our soak environment to get it some more exposure this week.
I've been trying with: But I only go a NLB without the EIPs deployed. Didn't find any errors in the logs. Do I have to enable a feature flag for it or kind of?
The reviewer found a typo where I may not be collecting the correct annotation. I'm waiting for the new build to complete and then rolling to our soak test. I'm passing through any AWS error and also error if the number of EIP and Subnets don't match. Let me get it online with the changes from code review and I'll dig in deeper. Also, are you using the binaries directly from the build system or just Kubernetes 1.12?
I checked with the fixes you provided yesterday. Works :) Created a service which spawns over 3AZs and supplied 3 EIP allocations. NLBs gets created correctly.
Can you also comment on the PR? I'm sure having multiple successful tests would be helpful to the reviewer.
Issues go stale after 90d of inactivity.

Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
Hey, thanks for doing this, I was just looking for a way to associate one eip to a nlb, one question though why do you need one eip for each AZ? now the nlb creates a 1 public IP that balance to 3 AZ using traefik in my case when I deploy the service, thanks
Due to the fact that EIPs are represented by network hardware in AWS's datacenter you'd like to have redundant instances. If you run with only one EIP and the AZ of this EIP goes down, you won't be able to reach the (still up and running) traefik deployment in the other two AZs.
Hi, I've seen the PR () has been waiting for an update for three months. do you need a hand? do you want me to pick it up? I am really interested on this to be merged :smile:
Can you help write the tests, ? I'm completely lost there to be honest.
Hi I've found out that test-coverage on the aws provider is low and there is no tests on the so I can't find tests on NLB. I am going to open a new issue and work on improving coverage for that and hopefully I'll find out how to add tests for the new annotation. As you can see below current coverage for and is low and can't find tests for NLB:
Thanks for the help. I thought I was just missing being able to locate them.
+1 need this ASAP We are hosted on EKS so only official version supported. Any idea when this will be merged?
.... As a workaround, I'm just manually creating a second NLB (with EIPs) and changing the listeners to forward the traffic to the automatically created target groups (from the original svc-linked NLB).
Waiting for this as well on EKS. Thanks!
would be great to have that in place. thanks! I can confirm that the manual workaround by works fine, but it is hard to automate that flow.
+1 Waiting for this as well on EKS.

+1
I have created a node project that runs as a cronjob and monitors a SOURCE target group and clones the targets to a DESTINATION target group. So when you create a LoadBalancer of type 'nlb' in kubernetes, all you need to do is manually create a new 'nlb' in AWS Console, assign your EIP to your NLB for a specific 'subnet'(availability zone), then create a target group with no targets; use the ARN from this target group as the destination and use the ARN of the kubernetes target group as the SOURCE. More information here: Public docker image:
I have hard time getting this working with NLB using ingress controller : Even subnets are not taking effect here , its not passing my configurations in the API that creates the NLB kind: Service apiVersion: v1 metadata: name: ingress-nginx namespace: ingress-nginx labels: ingress-nginx ingress-nginx annotations: nlb "eipalloc-,eipalloc-,eipalloc-" "subnet-,subnet-" spec: type: LoadBalancer selector: ingress-nginx ingress-nginx ports: name: http port: 80 targetPort: http name: https port: 443 targetPort: https =======================================================``
I am also getting the same error can anyone help me with this ??
what version of k8s you are using ? this is not supported before 1.15 as far as I know so my issue was that I tried to implement on EKS
Hi i am using the latest version of k8 1.15 .5
I am not using EKS with my solution but the problem is the same as you mentioned
New to EKS. Could you elaborate how the connectivity from NLB that we have created manually (as you mentioned ) to the ingress-nginx pod?
Has it worked for you in k8s 1.15 EKS?
How to get Allocation IDs?
my elastic eip ids cannot be found at service creation, so it just sits in pending forever. if I resort to listing the subnets on their own annotation, (not using subnet discovery) same thing. no matter what I do it seems to not find me elastic ip addresses. Are you sure nlb type is supported to work with Elastic IPs?

From the AWS docs it seem they are pretty heavily trying to discourage use of the nlb type (Cloud Provider Load Balancer) in favor of pushing the new AWS Load balancer.
See available annotations: https://kubernetes- If you have done that: what do your aws-lb-controller logs tell you? I guess there’s an access problem with your iam role. Is your aws-lb-controller iam autorized to list and modify eip’s? Are the eip annotations exactly correct, can they be found by the controller? The controller’s log stream should have some kind of an error there. I have successfully assigned eips using service annotations to provision an aws nlb, but pay attention, you need to install the aws-lb-controller yourself. The kubernetes built-in aws library does not support eips.
Here is the annotation needed for assigning a static Elastic IP to your NLB using the AWS Load Balancer (Ingress) Controller https://kubernetes- New Controller link https://kubernetes-


Question: go_test( name = "go_default_test", <del> srcs = ["mustrunas_test.go"], </del> <ins> srcs = ["capabilities_test.go"], </ins> importpath = "k8s.io/kubernetes/pkg/security/podsecuritypolicy/capabilities", library = ":go_default_library", deps = [

The name of this file is odd: The file context has nothing to do with Is this a BUG REPORT or FEATURE REQUEST?: /assign


Question: // the PATCH end-point. Returns true if the query param is supported by the // spec for the passed GVK; false otherwise. func supportsQueryParamV3(doc *spec3.OpenAPI, gvk schema.GroupVersionKind, queryParam VerifiableQueryParam) bool { <ins> if doc == nil || doc.Paths == nil { return false } </ins> for _, path := range doc.Paths.Paths { // If operation is not PATCH, then continue. op := path.PathProps.Patch

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Following the example here: Everything here works when you use , but when using 1.27.1, I see the following panic: What you expected to happen: When downgrading, this works as expected: How to reproduce it (as minimally and precisely as possible): <!-- Please make sure you are able to reproduce the bug using a supported version and version skew of Kubernetes. See for which versions and are currently supported. --You will need a Kubernetes cluster with Agones installed (I'm testing on 1.25.x, as you can see above), then follow this quickstart: Anything else we need to know?: The is implemented as an API Extension, not a CRD -- which is probably where the issue lies. Looking at the line: There seems to be an assuming that there will be a under and , in - there should be some nil checks in there. Environment: Kubernetes client and server versions (use ): See above Cloud provider or hardware configuration: GKE OS (e.g: ): Linux
seems like an another edge case; Due to the server version is old and openapiv3 is returning empty?
Thanks for opening this issue /triage accepted
I'm digging into the Agones code, but I'm 99.4% sure we don't even implement openapiv3 at all for our api endpoint
Yep, confirmed -- I see a bunch of 404's in my API Extension logs for requests for
/transfer kubernetes /kind regression
/sig cli /assign
I can also work on tomorrow if Sean can not get time to look at it
If nothing else, this does also force me to publish an openapi spec for our API extension
not sure if needed to handle nils inside the schema better, or if didn't fall back to v2 correctly in the absence of v3 data
It appears to be the former, where an OpenAPI V3 document is returned without , while the code assumes this field is non-nil.
seems like that should still fall back to v2
Can we get a kubectl -v8 to see what actually happens with downloading the OpenAPI?
As requested!

Thanks for the quick response. It looks like the OpenAPI V3 document is almost completely empty The code is expecting an Open API document with . I will have a fix out shortly.
I was digging through that - I found that really interesting. We only expose an basically empty spec for the api extension. It's always "just worked", so never needed to update it. I am curious where K8s is even getting the something that it is outputting for , but that's a side note.


Question: // PodToleratesNodeTaints checks if a pod tolerations can tolerate the node taints func PodToleratesNodeTaints(pod *v1.Pod, meta algorithm.PredicateMetadata, nodeInfo *schedulercache.NodeInfo) (bool, []algorithm.PredicateFailureReason, error) { <ins> if nodeInfo == nil || nodeInfo.Node() == nil { return false, []algorithm.PredicateFailureReason{ErrNodeUnknownCondition}, nil } if nodeInfo.Node().Spec.Unschedulable { return false, []algorithm.PredicateFailureReason{ErrNodeUnschedulable}, nil } </ins> return podToleratesNodeTaints(pod, nodeInfo, func(t *v1.Taint) bool { // PodToleratesNodeTaints is only interested in NoSchedule and NoExecute taints. return t.Effect == v1.TaintEffectNoSchedule || t.Effect == v1.TaintEffectNoExecute

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug I think this is a bug, but let me know if it is not. What happened: When feature-gate TaintNodesByCondition is true, we were surprised to find out that nodes that are continues to get pod scheduled onto it. What you expected to happen: Nodes that are don't get new pod scheduled. How to reproduce it (as minimally and precisely as possible): Set for kube-scheduler Drain a node and see pods continue to be scheduled to the node. Anything else we need to know?: From glancing the code, it appears that is removed when the feature gate is on. However, spec.Unschedulable appears to be checked only in Environment: Kubernetes version (use ): 1.9.2 Cloud provider or hardware configuration: baremetal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): 4.14.14 Install tools: custom Others:
/sig scheduling
ping I saw you touched this part of codes.
, good catch ! The options in my mind are Added a new predication for Unschedulable Taint nodes when cordon/uncordon
- From discussion with we might want to check node.Spec.Unschedulable when checking the taints ? (i.e on the predicate responsible for checking taints).
That's OK to me, just one comment that there maybe two same predicate error :).


Question: # TODO(thockin): uncomment this when we stop committing the generated files. #zz_generated.* zz_generated.openapi.go <ins> # TODO(roycaihw): remove this when we stop committing the generated definition !staging/src/k8s.io/apiextensions-apiserver/pkg/generated/openapi/zz_generated.openapi.go </ins> zz_generated_*_test.go # make-related metadata

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I'm trying to vendor , which due to (recently ?) code in depends on The problem now is that the content of is incomplete: apiextensions- calls generatedopenapi.GetOpenAPIDefinitions, but apiextensions-apiserver/pkg/generated/openapi/ doesn't contain the generated code. What you expected to happen: I expect to compile when fetching it with "go get" or vendoring it with tools like "dep". How to reproduce it (as minimally and precisely as possible): Environment: Kubernetes version (use ): master
/assign
/sig release
/assign
sent a fix in


Question: dc.rsListerSynced = alwaysReady dc.dListerSynced = alwaysReady dc.ssListerSynced = alwaysReady <ins> dc.recorder = record.NewFakeRecorder(10) </ins> informerFactory.Start(ctx.Done()) informerFactory.WaitForCacheSync(ctx.Done())

When using PDBs with or (percentage), it is not supported to reference pods via a PDB selector, that do not have a supported controller owner reference: Deployment, ReplicationController, ReplicaSet, and StatefulSet controllers or a resource implementing subresource This is also documented Nevertheless, there are still people who create PDBs a pods without controllers that ignore this. For example PDBs that set , want to block eviction, and node drain (it works for them but the PDB status is invalid). Unfortunately there is no feedback for the user to receive when this happens. The KCM logs is the only place that informs the user when unmanaged pods are found. Which I think is insufficient. There should be some way to inform users about this. Since it cannot be know on admission we probably have to do it when reconciling the PDB and unmanaged pods are found. Options: an event when this is detected - probably needs to be throttled in some way a condition in a PDB status that would be set and admins could monitor these to detect misbehaving PDBs create a PDB create unmanaged pod observe incorrect status because this configuration is unsupported No response <detailsNA </details<detailsNA </details<detailsNA </details<details</details<details</details<details</details
/sig apps
/triage accepted This was discussed during today's sig-apps, we've agreed to add a warning event right next to the log statement from the original description
I'll take this on.
/assign
/reopen I did some more testing for the unsupported controllers and found out that these are covered by the event, so I posted a followup to reflect that The new event should cover only the naked pods
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: ErrCodeResourceVersionConflicts ErrCodeInvalidObj ErrCodeUnreachable <del> ErrCodeTimeout </del> ) var errCodeToMessage = map[int]string{

various Oct 30 2023 6pm pst No response Looks like different test are failing with connection timeout errors: One of the tests triggered a core dump: No response /sig
are both suffers from pkg/registry and is from compoent-base, which likely same reason from `
/retitle failing-test: pkg/kubelet/eviction TestManagerWithLocalStorageCapacityIsolationOpen
/sig node
TestManagerWithLocalStorageCapacityIsolationOpen only failed in one of the PRs, so I don't think that specific test is the problem.
ok, rename as see fit. just put owning sigs please for triage.
TestStoreDeleteCollectionWithContextCancellation is the one that seems to be failing across all the failing PRs, so let's start there first. /retitle failing-test: TestStoreDeleteCollectionWithContextCancellation /sig api-machinery
triage link at shows this just starting (happening in CI jobs as well)
image used in the unit test job didn't change around the timeframe this was seen ()
looking for jumps in package unit test runtimes when this failure started happening to see if something is taking longer or stressing the system: ! - cc ! - cc ! - cc Could any of those have introduced changes that could be stressing or exhausting listener connections in unit tests?
the PR in question introduced a poll function around an API call that is unit tested with a fake client i.e. the poll is unit tested, but the fake client has reactors that return error or success right away, so in theory the poll should not be stretched to 1 retries with 1 connection attempts per fake client. while testing locally i did not notice any significant time difference after or the prior i can do some more checks to see what's going on.
merged right at the time the flakes were seen DeleteCollection invokes List, which was modified in that PR to cancel more aggressively cc
I can reproduce about a 10-20% flake rate at HEAD with: failures look like: or while testing that package with , I see ~1300-1500 open connections with this:
Let's revert it then. I looked into it and I don't know from a quick glance why it's failing and I don't have time to debug it now.

I opened
Okay, it looks like we need more debugging to open this PR
one optimization to strip some ~40 fake client connections and 2 seconds from a test:
sounds good, pointed at that commit as well
/triage accepted


Question: }) AfterEach(func() { <del> framework.ExpectNoError(framework.DeletePodWithWait(f, c, pod), "AfterEach: Failed to delete pod ", pod.Name) framework.ExpectNoError(framework.DeletePodWithWait(f, c, nfsServerPod), "AfterEach: Failed to delete pod ", nfsServerPod.Name) </del> <ins> err := framework.DeletePodWithWait(f, c, pod) Expect(err).NotTo(HaveOccurred(), "AfterEach: Failed to delete client pod ", pod.Name) err = framework.DeletePodWithWait(f, c, nfsServerPod) Expect(err).NotTo(HaveOccurred(), "AfterEach: Failed to delete server pod ", nfsServerPod.Name) </ins> }) // execute It blocks from above table of tests

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: See One /sig storage
/assign
[Slow] test suite was run. Glusterfs io stream test failed due to ns not being deleted, which was due to 1 pod remaining. The glusterfs-server pod was deleted. It appears that the glusterfs-client pod was not deleted even though returned no error. In cases where this test succeeds (which is most of the time) returns which is seen in the log as INFO: Ignore "not found" error above. Pod "gluster-io-client" successfully deleted. Sometimes when the plugin io streaming tests pass you'll see logged for the client pod. However, sometimes a client pod is deleted but the above log entry is not triggered. Eg in you'll see the nfs-client pod is deleted (the ns was able to be deleted) yet the above log msg stating to ignore the "not found" error was not generated. (scroll down to the line "I0724 20:27:20.364] Jul 24 20:27:00.559: INFO: Waiting for pod nfs-io-client in namespace 'e2e-tests-volume-io-pmdks' status to be 'terminated due to deadline exceeded'(found phase: "Running", readiness: true) (29.883170178s elapsed) I0724 20:27:20.365] [AfterEach] [sig-storage] NFS"). My guess is that in this case the plugin's client pod did not terminate in 30s -- WHY NOT THE FULL 5m? -- but did terminate in time for the ns to be deleted. And sometimes the client pod cannot be deleted in 30s (again WHY NOT THE FULL 5m?) and is still present when the ns is deleted. Note: if we cannot detect that the client pod was not actually deleted and then we delete the server pod, there is no chance that the ns can be deleted. Once the server pod is gone the client pod will not delete.
It looks like the client pod failed to terminate:
30s may not always be enough time for the gluster client pod to terminate?

also, why is it so common to see ~29 seconds for the client pods (nfs, gluster) to supposedly terminate. The wait should be for up to 5m yet I never see = 30s wait times... yet, we know that sometimes the gluster client pod is not terminated. Seems like maybe a bug in this area?
can you take a look? My findings so far are this test fails about 20% of the time. The gluster client pod's Status is set to Failed and therefore the pod won't be deleted. There is e2e code that waits for the pod to be deleted but escapes if the pod has "Failed" and if the passed-in reason matches the pod's Status.Reason. In the gluster pod test failures the pod's Status.Reason == "". This causes the test to incorrectly assume that the pod was deleted when it wasn't. This problem manifests itself when the namespace is deleted and that step fails. It seems like the Failed pod's Status.Reason should not be blank. cc
It looks like gluster unmount failed:
But in the log I looked at today (https://k8s-) I see the pod's Status=Running. Then the pod is deleted and the status remains Running for ~30s then changes to Failed. Even though the pod is Failed its Reason remains "". So 1) I wonder why the pod is marked as Running if its mount failed, and 2) once the pod is Failed why is Reason still blank?
It was the unmount that failed, not the mount.
ah, ok thanks, I missed that. Still need to know why Pod.Status.Reason is blank...
Below are some records from a related to the glusterfs-client pod, gluster-io-cliente2e-tests-volume-io-6h140(-778e-11e7-afad-)_ and the "e2e-tests-volume-io-6h140" ns:
I think the issue is that the test is deleting the gluster endpoints before deleting the pod.
Nm, the test does seem to be cleaning up in the correct order: client pod, then server pod and endpoints. So the issue seems to be that during Pod termination, it goes to Failed state.

Before, I thought it was because the unmount failed, but based on the timestamps of the logs, it seems like it fails before we start unmounting.
Just spoke to and he said that it's normal for a Pod to go to Failed state during the termination sequence. In between Failed state and Pod object deletion is when we unmount the volumes. So I think the issue here is that the WaitForPodTerminated function needs to wait only for the Pod object to be deleted, and not return early if Pod state is Failed.
do you have time to look at this? The test is still flaking
yes I can take a look. Do you know if there has been a pr started to address this issue? BTW, I will be gone most of tomorrow but will start tomorrow night.


Question: glog.Warningf("no LUN available for instance %q (%v)", nodeName, err) return "", fmt.Errorf("all LUNs are used, cannot attach volume %q to instance %q (%v)", volumeSource.DiskName, instanceid, err) } <del> glog.V(4).Infof("Trying to attach volume %q lun %d to node %q.", volumeSource.DataDiskURI, lun, nodeName) </del> <ins> glog.V(2).Infof("Trying to attach volume %q lun %d to node %q.", volumeSource.DataDiskURI, lun, nodeName) </ins> isManagedDisk := (*volumeSource.Kind == v1.AzureManagedDisk) err = diskController.AttachDisk(isManagedDisk, volumeSource.DiskName, volumeSource.DataDiskURI, nodeName, lun, compute.CachingTypes(*volumeSource.CachingMode)) if err == nil { <del> glog.V(4).Infof("Attach operation successful: volume %q attached to node %q.", volumeSource.DataDiskURI, nodeName) </del> <ins> glog.V(2).Infof("Attach operation successful: volume %q attached to node %q.", volumeSource.DataDiskURI, nodeName) </ins> } else { glog.V(2).Infof("Attach volume %q to instance %q failed with %v", volumeSource.DataDiskURI, instanceid, err) return "", fmt.Errorf("Attach volume %q to instance %q failed with %v", volumeSource.DiskName, instanceid, err)

<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Current logging is not enough for azure disk diagnostics, change some log level from 4 to 2 What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: /sig azure /assign


Question: make -C "${KUBE_ROOT}" WHAT=cmd/kubeadm make -C "${KUBE_ROOT}" test  <del> WHAT=k8s.io/kubernetes/cmd/kubeadm/test  </del> <ins> WHAT=k8s.io/kubernetes/cmd/kubeadm/test/cmd  </ins> KUBE_TEST_ARGS="--kubeadm-path '${KUBEADM_PATH}'"

go test ./cmd/kubeadm/... fails with: it appears to be following a "RunCmd" pattern used by e2e tests, but unit tests should not require built binaries
can work on this
has a long discussion as to why this is. These tests only get run when gets run, which will produce the needed binary and pass it in as arg. For more information, authored this and should be able to provide more details.
Yup, the target explicitly ignores these tests: and like said, runs this script as part of its target:
then the test should have build tags to exclude it from a regular run of the package
or up for fixing that?
I thought build tags only works on the OS level or if you it would ignore it completely? Wouldnt break the command as well?
I'd also tolerate skipping the test if the built binary doesn't exist
I think that is a great solution. I think in I had initially suggested something like that. If everyone is ok with that, I will go ahead and make that change.
Sounds good, make that change and we'll see how it turns out. I'd really want these tests start working, so +1


Question: import ( "errors" "fmt" <del> "math/big" "net" </del> api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/registry/core/service/allocator" <del> "k8s.io/utils/integer" utilnet "k8s.io/utils/net" </del> <ins> "math/big" "net" </ins> ) // Interface manages the allocation of IP addresses out of a range. Interface

What happened: All the services are asking to be recreated because the service CIDR changed after upgrading from 1.16.3 ro 1.17.0 What you expected to happen: Smooth migration from to , which would not break service CIDR. How to reproduce it (as minimally and precisely as possible): Do an upgrade from version 1.16.3 to 1.17.0 Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ):Kernel (e.g. ):Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
do you mean that the service cidr changed without any intervention after the upgrade? EDIT 96.0.0/12 is the default service cidr when using kubeadm --service-cidr string Default: "10.96.0.0/12" EDIT if not using kubeadm the default is May it be possible you were using another service cidr before the upgrade and you didn't set it during the upgrade, so it fall back to the default?
Can you provide your kube-apiserver invocation?
xref which warnings if no explicit CIDR flag was passed but should not have changed behavior otherwise
Where do you see that? The default is specified here as 10.0.0.0/24 and did not change between 1.16.0 and 1.17.0:
I was assuming it was using kubeadm based on the log message 96.0.0/12 is the default service CIDR for kubeadm
ah, ok. If they are using kubeadm, that default also did not change between 1.16.0 and 1.17.0:
a few questions for you: how did you install your 1.16.x cluster? did you customize the service cidr in that deployment? how did you upgrade your 1.16.x cluster to 1.17.x? did you apply the same service cidr customization in that deployment?
16.x was a straight and normal install no change on service CIRD, and we have never touched this CIDR configuration. And kubeadm is supposed to be used during the upgrade from 1.16.x to 1.17.x.

We just followed the describe process: The logs system started showing the shared message.
can you reproduce?
can you paste ? That will show you kubeadm config
created a single node cluster with v1.16.4 and a custom service cidr: applied upgrade to v1.17.0: i do not see a change in the custom service cidr.
if you never touched the service CIDR on the initial install it should have been which is the kubeadm default. your config map shows . how did you adapt the IP to work with the kubeadm default service cidr () if you initially deployed without modifying said cidr? to me this suggests that a change was required at some point.
The only change is the update from 1.16.3 to 1.17.0, done yesterday. Please find below a snapshot from our logs: <img width="1635" alt="Screen Shot 2019-12-21 at 22 22 29" src="https://user-"
my point is that if you did not apply a custom service cidr on "kubeadm init", then this cidr was the default out of kubeadm and after upgrade the service cidr was preserved as your config map shows. what was the elasticsearch/logging service IP before upgrade? EDIT: also do you have the YAML that was used to create this service?
101.249.43 belongs to the service rubnet range at minimum the error message is wrong /kind bug
are those errors happening constantly or only happened during the upgrade? are you able to create new services within that range? let's say, with a service IP of 10.101.249.45 Also, can you paste your current service-cluster-cidr configured in the controller-manager and apiserver? i.e.
this is a bug in core: this call in the repair controller returns for an IP that is actually in the CIDR range: also indicates a missing unit test case. latest commits: this might have regressed with the dual-stack changes? cc /remove-triage support needs-information /sig network /remove-sig cluster-lifecycle
/priority important-longterm
I was able to reproduce the same behavior again, with a fresh install k8s release 1.16.4.

Just After installing metrics server, we could notice a shift in the service CIDR, but nothing in the logs.
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry.
/cc
so the problem here is that an IPv4 CIDR such as translates to addresses. but this line here clamps the maximum to 2^16 () so for an address like the logic in the ip allocator fails in the function: as the offset ends up bigger than the maximum. it feels like the maximums for IPv4 and IPv6 should be different and larger. otherwise for CIDRs in the notation : can give incorrect results for IPv4 can give incorrect results for IPv6 cc who touched related logic.
I think that was touched in … was this bug preexisting or did that change behavior?
before PR () i do not see a clamp for IPv4: only one for IPv6 at 16bits.
We should: revert add unit tests for the missed cases get that into 1.17.1 reintroduce the reliance on the function carefully
What would be the safest path to recreate all the services whilst waiting a final solution ? Especially for production K8S.
to mitigate the error in 1.17.0 you need to modify the serivce CIDR to be , instead of . modify the ConfigMap CIDR to have modify the kubeadm static pod manifest for the api server () to have edit problematic Service objects and give them an IP in the range if needed.
Thank you for those instructions, I'm affected by this, does this have anything to do with a possibility of ClusterIP's not properly responding or does it only print out alerts?
hello, can you test if the ClusterIP is not responding? i don't think we got a confirmation if this is only an alert. my guess is that it is just that, but there might be side effects.
i see you logged i will comment there.
How do you "recreate" a service? Literally delete and create again? Or is there a command which would cause it to re-create itself without having to be deleted?


Question: PATH="${GOBIN}:${PATH}" # Install tools we need, but only from vendor/... <del> go install ./vendor/github.com/client9/misspell/cmd/misspell </del> <ins> go install k8s.io/kubernetes/vendor/github.com/client9/misspell/cmd/misspell </ins> # Spell checking # All the skipping files are defined in hack/.spelling_failures

I am trying to update aws-sdk-go dep version, following the guide in I checkout to the master branch, and I use these commands: But all the "BUILD" files in the "vendor" folder are deleted. What did i do wrongWhat are the right steps? When I use the same commands in the "release-1.9" branch, all these files are not changed.
/sig contributor-experience
I came across this recently as well. same for me. /kind bug
Looks like is the culprit? /cc
you didn't do anything wrong. :) Updating bazel files again will solve the problem for now: But we should either fix the script or update the instructions.
This should run already as a part of godep- Is that step producing any errors?
There are not any error message. Here is the output:
yeah, I can reproduce this - running yields lots of missing files. still investigating, but I'm guessing the files are being generated in the generated gopath (from , rather than the repo root.
... though that shouldn't matter, since it's just a symlink. hm.
I think I found it.. I think it has to do with the fact that the init script uses to strip out symlinks, which doesn't play nice with . I'm testing a fix locally.. if it works, I'll push a PR. I don't know if removing the flag will break other things tho.
Nope. That didn't work. Next test I'm running is a revert of .
after running , if I then run , it still doesn't regenerate the files, which makes me think doesn't like running in the artificial gopath.
oh, I think might full resolve either the or its working directory, which causes problems since we have an artificial gopath. cc if he's seen this before.
seems to need to run from the real (or at least a subdir of the real ). seems to need to run from a valid - i.e. must be in . If I to the real in , works properly, but then explodes.
Just a straight revert of that last change didn't work either. I wonder how long this has been broken for..
When Gazelle starts, it locates the repository root (if is not given, it's the nearest parent directory containing a file named ), then evaluates symlinks in that path.

When Gazelle walks the directory tree, it only follows symlinks to directories that point outside the repository. It checks that by evaluating symlinks, then checking whether the resolved path has the resolved repository root as a prefix. This is all in service of avoiding symlink cycles and preventing the same directory from being visited on more than one path (which would cause incorrect or inconsistent attributes).
I'd guess since .
So I think passing to Gazelle would fix it. Then it won't go looking outside the GOPATH for the WORKSPACE file.
alone doesn't seem to be helping, at least not if I set it to the real repo root. and the files under are still missing. What does seem to help is if I to the real repo root before calling .
note also that it seems that gazelle isn't walking any of the tree when run under ; it's not just that's affected.
Easiest way to reproduce: out kubernetes/kubernetes some file in the tree (e.g. ) that the file you removed wasn't regenerated that the file was now regenerated
yeah, gazelle really doesn't like the symlink. super hacky fix that is probably not a good idea here:
Copying my comment from slack:
Please try again with a fresh master. Fix was merged in last night.
It's ok now!


Question: CLEANUP=true GINKGO_FLAGS='--skip="[Flaky]|[Serial]"' SETUP_NODE=false <del> TEST_ARGS=--cgroups-per-qos=true </del> <ins> TEST_ARGS=--cgroups-per-qos=false </ins>

In node-e2e-test on Containervm image, the containers of the pods created by Kubernetes are subcontainers of docker-daemon/docker/. It means that the runtime resource usage (docker-daemon) is not correct, because all pods are included. This problem maybe due to container misconfiguration in node-e2e-test.
The kubelet on debian-based containervm image is misconfigured. Will fix it shortly.


Question: ) // RcByNamePort returns a ReplicationController with specified name and port <del> func RcByNamePort(name string, replicas int32, image string, port int, protocol v1.Protocol, </del> <ins> func RcByNamePort(name string, replicas int32, image string, containerArgs []string, port int, protocol v1.Protocol, </ins> labels map[string]string, gracePeriod *int64) *v1.ReplicationController { return RcByNameContainer(name, replicas, image, labels, v1.Container{ Name:  name, Image: image, <ins> Args:  containerArgs, </ins> Ports: []v1.ContainerPort{{ContainerPort: int32(port), Protocol: protocol}}, }, gracePeriod) }

<!-- Please only use this template for submitting reports about failing tests in Kubernetes CI jobs --Which jobs are failing: ci-kubernetes-e2e-gci-gce-serial Which test(s) are failing: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes Since when has it been failing: These tests have been failing since 6/28 Failures coincide with the following changes: Testgrid link: Reason for failure: The failure messages look as follows: [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to add nodes [sig-cluster-lifecycle] Nodes [Disruptive] Resize [Slow] should be able to delete nodes /cc /milestone v1.16 /priority critical-urgent /kind failing-test /cc
/sig cluster-lifecycle
For more context, the code for the failing tests is here The error for both comes through  - some Pod is not responding. It is failing to find pods prefixed with "my-hostname-add-node". Within the logs we can see this: So the Pods are stuck in a Pending state. The Pods are supposed to be created by a ReplicationController in , which itself uses this method from the e2e framework . However, after being in a Pending state (after the "failed to wait for pods responding: timed out waiting for the condition" error is registered), they do seem to reach a Ready state
/assign for triage.
I think agnhost needs to be passed , but did not do so in /assign sig-testing
GitHub didn't allow me to assign the following users: sig-testing. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : I think agnhost needs to be passed , but did not do so in Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Reopen to monitor job /reopen
Reopened this issue.

<detailsIn response to : Reopen to monitor job /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thank you for working on the PR to solve this. indeed solved the issue reported here. /close
Closing this issue. <detailsIn response to : Thank you for working on the PR to solve this. /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details


Question: podRequest += value } <ins> for i := range pod.Spec.InitContainers { initContainer := &pod.Spec.InitContainers[i] value := schedutil.GetNonzeroRequestForResource(resource, &initContainer.Resources.Requests) if podRequest < value { podRequest = value } } </ins> // If Overhead is being utilized, add to the total requests for the pod if pod.Spec.Overhead != nil && utilfeature.DefaultFeatureGate.Enabled(features.PodOverhead) { if quantity, found := pod.Spec.Overhead[resource]; found { podRequest += quantity.Value() } } return podRequest }

The scheduler does not properly calculate pod requests for a resource as - it ignores init containers completely in . This would allow a workload to get "free" scheduling above the limit of containers (mitigated by quota, which calculates this properly) and so the scheduler would make non-optimal fits. Both kubelet and scheduler enforce the , so you can't actually schedule, but you sometimes get steered to the wrong node if you have more init container resource usage than containers. in should use and the similar helpers instead of writing its own method to calculate the request for resources. /sig scheduling
/priority important-soon
Let me follow this issue. /assign
fyi
After I checked the implementation of function, I think we just need to use it to instead of the function in the scheduler.
/priority critical-urgent
Is the pod phase not considered? When the pod phase is running, the requested resource is actually considered as . This change seems to reduce the resource utilization of the node.
Actually, I don't think we need to consider the pod.Phase. The resource request & limit are mainly used in the scheduling process. The scenario you said is not belong to the responsibility of kube-scheduler
Correct. Setting appropriate requests/limits that maintain good node usage is users responsibility. Considering the max(initContainers, sum(containers)) is the documented behavior.
I understand. Thank you for your comments!


Question: import ( "k8s.io/kubernetes/pkg/api" apierrors "k8s.io/kubernetes/pkg/api/errors" <ins> errs "k8s.io/kubernetes/pkg/util/errors" </ins> ) <del> // NewForbidden is a utility function to return a well-formatted admission control error response func NewForbidden(a Attributes, internalError error) error { // do not double wrap an error of same type if apierrors.IsForbidden(internalError) { return internalError } name := "Unknown" kind := a.GetKind() </del> <ins> func extractKindName(a Attributes) (name, kind string, err error) { name = "Unknown" kind = a.GetKind() </ins> obj := a.GetObject() if obj != nil { objectMeta, err := api.ObjectMetaFor(obj) if err != nil { <del> return apierrors.NewForbidden(kind, name, internalError) </del> <ins> return "", "", err </ins> } // this is necessary because name object name generation has not occurred yet

Currently the admission controller returns a forbidden (403) error if the namespace doesn't exist. We should return a 404 error instead.


Question: } run := func(t *testing.T, test testcaseType) { <del> // Create StorageClasses </del> <ins> t.Log("Creating StorageClass") </ins> suffix := rand.String(4) classes := map[string]*storagev1.StorageClass{} classes[classImmediate] = makeDynamicProvisionerStorageClass(fmt.Sprintf("immediate-%v", suffix), &modeImmediate, nil)

Which jobs are failing: pull-kubernetes-integration Which test(s) are failing: TestVolumeProvision Example: Since when has it been failing: Not sure, and aren't displaying any results /sig storage
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/assign /remove-lifecycle rotten Still seeing this, although not very frequently. Could indicate a race.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
latest failures:
take as an example the test fails b/c is set to PVC , which is not expected. our volume binder tries to dynamically provision it:
as for the test fails for the same reason when the scheduler volume binder tries to find volumes for PVC , its corresponding volume is not synced to volume binder or synced version is not in Available state so the volume binder will try to dynamically provision it too
I think the root cause is that the scheduler fails to find matching PVs in PV Assume Cache for the PVC because the PV in the Assume Cache is not in Available state or does not exist due to event delay. We should wait for the PV/PVC Assume Cache to be up to date before scheduling the pod.

However, it's not possible in integration tests. I think we can wait for the PV to be available or bound. This should reduce failures.


Question: - Removed cadvisor metric labels `pod_name` and `container_name` to match instrumentation guidelines. Any Prometheus queries that match `pod_name` and `container_name` labels (e.g. cadvisor or kubelet probe metrics) must be updated to use `pod` and `container` instead. ([#80376](https://github.com/kubernetes/kubernetes/pull/80376), [@ehashman](https://github.com/ehashman)) <del> ### Depreciated/changed metrics </del> <ins> ### Deprecated/changed metrics </ins> - kube-controller-manager and cloud-controller-manager metrics are now marked as with the ALPHA stability level. ([#81624](https://github.com/kubernetes/kubernetes/pull/81624), [@logicalhan](https://github.com/logicalhan)) - kube-proxy metrics are now marked as with the ALPHA stability level. ([#81626](https://github.com/kubernetes/kubernetes/pull/81626), [@logicalhan](https://github.com/logicalhan))

What happened: There's a typo typo in and What you expected to happen: “depreciated” should be “deprecated”. How to reproduce it (as minimally and precisely as possible): View the release notes Anything else we need to know?: /lifecycle active I have a PR to fix this /sig docs Environment: Kubernetes version (use ): v1.16 Cloud provider or hardware configuration: not applicable OS (e.g: ): not applicable Kernel (e.g. ): not applicable Install tools: not applicable


Question: var _ Interface = new(cadvisorUnsupported) <del> func New(port uint, runtime string, rootPath string) (Interface, error) { </del> <ins> func New(address string, port uint, runtime string, rootPath string) (Interface, error) { </ins> return &cadvisorUnsupported{}, nil }

cAdvisor provides potentially sensitive data and there's currently no way to block access with anything other than iptables.
What type of cluster are you running (GKE, or what instructions did you use to setup your cluster?) A short term solution might be to try adding flag to kubelet and restarting the kubelet service and see if the sensitive information is no longer leaked. Doing this may the UI and heapster which expect to be able to poll the cadvisor port. I think probably knows if there is a longer term solution planned, such as: only serve on specified interface. if this is localhost, then it breaks the ui, but maybe that is okay. maybe has thought about this. require authentication/authorization to connect to cAdvisor port.
Self deployed cluster, the kubelet is started via
We are working on improving the kubelet APIs which will subsume cadvisor APIs. Until then setting cadvisor port to '0' is probably the way to go.
Is there an issue and target date for doing this?
Filed . The current plan is to complete this for v1.1. cc
Do you mean the read-only kubelet port ( by default IIRC)? If so, then we really need to disable this & provide authorisation options on the kubelet to be able to retrieve stats fro the kubelet via the normal port with a restricted token/cert/auth options. We've discussed this for OpenShift (openshift/origin) too.
Sorry I didn't realise that cadvisor was bound to port 4194 as well. Thought this was all embedded inside the kubelet process without exposing any ports.
Any reason why we start up cadvisor http server at all? Seems unnecessary now kubernetes embeds cadvisor & exposes via & ?
does not expose all the containers (, , etc). is working on introducing a better stats API in kubelet () that should help us shut down cadvisor's http server in kubelet.
Couldn't we just add cadvisor as a handler in the kubelet anyway? No need for separate http server & would mean access to raw data is available if anyone wants it.
Indeed that is an option. But we need a better stats API in kubelet anyways for reasons mentioned in .
Absolutely agree but thinking quick win.
+1 on a quick win.

We just disabled it by default instead: Which reminds me it would be great to have so we could make it easy to turn on/off stuff like this w/o rewriting the entire command-line in a service file.
cc
it's not 100% clear to me whether or not made it safe to use , could you please clarify? On a fresh k8s 1.6.4 cluster still exposes cAdvisor to the world: the file you referenced was deleted as part of coreos/coreos-overlay the documentation @ (CoreOS docs) doesn't mention cAdvisor should be disabled either. Another reason for my question to Vish.
Ahh. That's a bug that the new component config based scheme introduced. being is a valid setting.
I think we can change this. The KubeletConfiguration type is still technically alpha, so it should be ok to make this field a pointer and keep the current default when it's not provided (but allow 0). This won't affect anyone who presently omits the flag and expects port 4194. Theoretically people could be explicitly passing 0 and expecting 4194... I guess... but that's sort of an insane case, and I'd consider that behavior a bug.
Thanks for clarifying guys :+1: The original problem remains though: enabling the cAdvisor port makes the kubelet listen on all interfaces.
Yeah. We can get cAdvisor to bind to the interface that kubelet uses. cc , Antoine Cotten <:
/assign


Question: "encoding/json" "fmt" "math" <ins> "sync" </ins> "time" "github.com/onsi/ginkgo"

From After , Multi-AZ Clusters tests in 1.19 and 1.20 started timing out as createBalancedPodForNodes is extremely inefficient in scale: for each node (out of 5k), it lists all pods in a cluster serially (each list takes ~5s), which translates to ~7 hours. Optimizing this function by listing pods once and then in memory by Spec.NodeName /kind bug /priority critical-urgent /sig scheduling
/triage accepted
/assign
/reopen Looks that ComputeCPUMemFraction took only 12 minutes, but it stuck later, on It is scheduling one pod per each node: Which is waiting 5s to be ready: 5k nodes * 5s = ~7h
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Can you do the pod creations in parallel?
Yes, good idea. I'll update to do that


Question: "//staging/src/k8s.io/apimachinery/pkg/util/uuid:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", "//staging/src/k8s.io/client-go/kubernetes:go_default_library", <ins> "//staging/src/k8s.io/client-go/util/exec:go_default_library", </ins> "//test/e2e/framework:go_default_library", <ins> "//test/e2e/framework/log:go_default_library", </ins> "//test/e2e/framework/node:go_default_library", "//test/e2e/framework/pod:go_default_library", "//test/e2e/framework/ssh:go_default_library",

<!-- Please only use this template for submitting enhancement requests --What would you like to be : separate stdout and stderr command exit code: remove kubectl dependency: Why is this needed: to replace
/assign
/sig storage HostExec was originally written for local PV tests. If we want to replace SSH in e2e, some features must be supported and I’d like to help on this. cc


Question: t := reflect.TypeOf(obj).Elem() gvk := version.WithKind(t.Name()) s.unversionedTypes[t] = gvk <del> if _, ok := s.unversionedKinds[gvk.Kind]; ok { panic(fmt.Sprintf("%v has already been registered as unversioned kind %q - kind name must be unique", reflect.TypeOf(t), gvk.Kind)) </del> <ins> if old, ok := s.unversionedKinds[gvk.Kind]; ok && t != old { panic(fmt.Sprintf("%v.%v has already been registered as unversioned kind %q - kind name must be unique", old.PkgPath(), old.Name(), gvk)) </ins> } s.unversionedKinds[gvk.Kind] = t }

We have moved to non-global schema in a number of places in 1.6 and 1.7. It is easy to forget registering the unversioned types like , leading to encoding errors in error cases. The registration of the unversioned types should be automatic as for the types, i.e. in .
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
do we need to close out this issue? (since referenced PR has merged)
Fixed by


Question: if err != nil { return nil, fmt.Errorf("failed to decode managed fields: %v", err) } <del> // We can assume that patchObj is already on the proper version: // it shouldn't have to be converted so that it's not defaulted. // TODO (jennybuckley): Explicitly checkt that patchObj is in the proper version. </del> <ins> // Check that the patch object has the same version as the live object patchObj := &unstructured.Unstructured{Object: map[string]interface{}{}} if err := yaml.Unmarshal(patch, &patchObj.Object); err != nil { return nil, fmt.Errorf("error decoding YAML: %v", err) } if patchObj.GetAPIVersion() != f.groupVersion.String() { return nil, errors.NewBadRequest( fmt.Sprintf("Incorrect version specified in apply patch. "+ "Specified patch version: %s, expected: %s", patchObj.GetAPIVersion(), f.groupVersion.String())) } </ins> liveObjVersioned, err := f.toVersioned(liveObj) if err != nil { return nil, fmt.Errorf("failed to convert live object to proper version: %v", err)

What happened: When sending a request to the server side apply endpoint (), if the apiVersion provided in the object is different from the apiVersion in the url, it returns a 500 error, like: What you expected to happen: The request should fail if the apiVersion provided in the object is different from the apiVersion in the url, but it should fail before trying to merge, and be a 4xx error. Probably a bad request error. Anything else we need to know?: The check should probably be done somewhere near this . The issue, though, is that the patch is given as a byte array, and needs to be deserialized into an object to access the apiVersion field. How to reproduce it (as minimally and precisely as possible): With a kube-apiserver built from head, running with :
/kind bug /sig api-machinery /wg apply /priority important-longterm /assign


Question: metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/util/wait" clientset "k8s.io/client-go/kubernetes" <del> extensionsinternal "k8s.io/kubernetes/pkg/apis/extensions" </del> "k8s.io/kubernetes/pkg/client/clientset_generated/internalclientset" deploymentutil "k8s.io/kubernetes/pkg/controller/deployment/util" <del> "k8s.io/kubernetes/pkg/kubectl" </del> testutils "k8s.io/kubernetes/test/utils" )

As the title said, we need update touse server side reaper. xref:
/sig apps
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
/assign


Question: apiVersion: extensions/v1beta1 kind: DaemonSet metadata: <del> name: fluentd-gcp-v1.34 </del> <ins> name: fluentd-gcp-v1.35 </ins> namespace: kube-system labels: k8s-app: fluentd-gcp kubernetes.io/cluster-service: "true" <del> version: v1.34 </del> <ins> version: v1.35 </ins> spec: template: metadata: labels: k8s-app: fluentd-gcp kubernetes.io/cluster-service: "true" <del> version: v1.34 </del> <ins> version: v1.35 </ins> spec: containers: - name: fluentd-gcp <del> image: gcr.io/google_containers/fluentd-gcp:1.34 </del> <ins> image: gcr.io/google_containers/fluentd-gcp:1.35 </ins> # If fluentd consumes its own logs, the following situation may happen: # fluentd fails to send a chunk to the server => writes it to the log => # tries to send this message to the server => fails to send a chunk and so on.

<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): no What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): fluentd timezone timeformat Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): What happened: Log lines from docker have the wrong timestamp. Anything else do we need to know: We are building a custom Docker Image for fluentd/td-agent which is based on your Dockerfile but with some slight changes. One of which is that we set the timezone to something else than UTC. In this case fluentd will produce wrong timestamps for log lines since the string only includes a simple Z in the end. Changing the time_format to fixed the problem for us. Of course this is not really a bug in kubernetes but changing the default to the above will probably help others in the future.
Thanks for reporting! It seems to be a bug


Question: <!-- TODO: Update, clean up. --> Mesos allows dynamic sharing of cluster resources between Kubernetes and other first-class Mesos frameworks such as [Hadoop][1], [Spark][2], and [Chronos][3]. <del> Mesos also ensures applications from different frameworks running on your cluster are isolated and that resources are allocated fairly. </del> <ins> Mesos also ensures applications from different frameworks running on your cluster are isolated and that resources are allocated fairly among them. </ins> <del> Running Kubernetes on Mesos allows you to easily move Kubernetes workloads from one cloud provider to another to your own physical datacenter. </del> <ins> Mesos clusters can be deployed on nearly every IaaS cloud provider infrastructure or in your own physical datacenter. Kubernetes on Mesos runs on-top of that and therefore allows you to easily move Kubernetes workloads from one of these environments to the other. </ins> This tutorial will walk you through setting up Kubernetes on a Mesos cluster. <del> It provides a step by step walk through of adding Kubernetes to a Mesos cluster and running the classic GuestBook demo application. The walkthrough presented here is based on the v0.4.x series of the Kubernetes-Mesos project, which itself is based on Kubernetes v0.11.0. </del> <ins> It provides a step by step walk through of adding Kubernetes to a Mesos cluster and starting your first pod with an nginx webserver. </ins> <del> **NOTE:** There are [known issues with the current implementation][11]. Please [file an issue against the kubernetes-mesos project][12] if you have problems completing the steps below. </del> <ins> **NOTE:** There are [known issues with the current implementation][7] and support for centralized logging and monitoring is not yet available. Please [file an issue against the kubernetes-mesos project][8] if you have problems completing the steps below. </ins> ### Prerequisites <del> * Understanding of [Apache Mesos][10] * Mesos cluster on [Google Compute Engine][5] * A VPN connection to the cluster. </del> <ins> * Understanding of [Apache Mesos][6] * A running [Mesos cluster on Google Compute Engine][5] * A [VPN connection][10] to the cluster * A machine in the cluster which should become the Kubernetes *master node* with: * GoLang > 1.2 * make (i.e. build-essential) * Docker **Note**: You *can*, but you *don't have to* deploy Kubernetes-Mesos on the same machine the Mesos master is running on. </ins> ### Deploy Kubernetes-Mesos <del> Log into the master node over SSH, replacing the placeholder below with the correct IP address. </del> <ins> Log into the future Kubernetes *master node* over SSH, replacing the placeholder below with the correct IP address. </ins> ```bash ssh jclouds@${ip_address_of_master_node}

Hi We are updating docs to v1beta3. v1beta1 and v1beta2 will be abandoned soon. For the examples in getting-started-guides, we tend to use hyperlinks to point to examples instead of letting them have their own individual examples (see ). However, in the mesos guide, the examples contain mesos specific setup and knowledge, so I can't simply remove the texts and point to kubernetes/examples/guestbook, which is updated to v1beta3. My suggestion is that we update the guestbook example in kubernetes-mesos, and use a hyperlink in mesos getting-started-guide to replace the example texts. Could you help to make the changes? Thanks.
cc
We're going to delete v1beta1 and v1beta2 in O(days). See PR .
cc
we've initiated an effort to upstream some components into k8s. as part of that effort, we're upgrading the guestbook examples for k8s-mesos: the getting started guide for mesos will likely have to change in order to align with the move upstream. tracking this issue here: , Brian Grant wrote:
xref , James DeFelice wrote:
Ref
cc
xref
This needs to be moved to the v1 API.
Agreed. We have changes in review right now, should be merged soon. , Brian Grant wrote:


Question: echo "Generating protobuf requires protoc 3.0.0-beta1 or newer. Please download and" echo "install the platform appropriate Protobuf package for your OS: " echo <del> echo "  https://github.com/google/protobuf/releases" </del> <ins> echo "  https://github.com/protocolbuffers/protobuf/releases" </ins> echo echo "WARNING: Protobuf changes are not being validated" exit 1

google protobuf repository has moved from old location pointing currently to Point scripts to the new location ,kubernetes-csi/csi-lib-utils,kubernetes/kubernetes N/A Kubernetes version (use ): 1.21 Cloud provider or hardware configuration: N/A OS (e.g: ): Mac OS Kernel (e.g. ): Darwin Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
FYI
/assign
/area code-organization /kind cleanup


Question: # $1: The name of the instance. # $2: The scopes flag. # $3: The minion start script. <del> function create-instance { </del> <ins> function create-minion { </ins> detect-project local attempt=0 while true; do

The node disk type should be an option in the cluster/gce/config-*.sh files. The nodes get created here:
And also the disk size.
How to do this for aws ? I am stuck at 10GB minion
It . A pull request adding it would be very welcome!
please check this thread
Hello there, can I do something to make this available in the command ? I'd like to us an SSD drive :)


Question: // This returns the full identifier of the primary NIC for the given VM. func (ss *scaleSet) getPrimaryInterfaceID(machine compute.VirtualMachineScaleSetVM) (string, error) { <ins> if machine.NetworkProfile == nil || machine.NetworkProfile.NetworkInterfaces == nil { return "", fmt.Errorf("failed to find the network interfaces for vm %s", to.String(machine.Name)) } </ins> if len(*machine.NetworkProfile.NetworkInterfaces) == 1 { return *(*machine.NetworkProfile.NetworkInterfaces)[0].ID, nil } for _, ref := range *machine.NetworkProfile.NetworkInterfaces { <del> if *ref.Primary { </del> <ins> if to.Bool(ref.Primary) { </ins> return *ref.ID, nil } } <del> return "", fmt.Errorf("failed to find a primary nic for the vm. vmname=%q", *machine.Name) </del> <ins> return "", fmt.Errorf("failed to find a primary nic for the vm. vmname=%q", to.String(machine.Name)) </ins> } // getVmssMachineID returns the full identifier of a vmss virtual machine.

<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: kube-controller-manager panic when network interfaces for Azure VMSS are null. This happens when Azure cloud provider tried to get IP address for VMSS VM: What you expected to happen: kube-controller-manager should report an error instead of panic. How to reproduce it (as minimally and precisely as possible): Scale cluster nodes up and down frequently. Anything else we need to know?: /kind bug /area provider/azure Environment: Kubernetes version (use ): 1.15+ Cloud provider or hardware configuration: Azure OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cloud-provider /assign


Question: fmt.Printf("--output wide is not available in kubectl config view; reset to default output format (%s)nn", defaultOutputFormat) cmd.Flags().Set("output", defaultOutputFormat) } <ins> if outputFormat == "" { fmt.Printf("reset to default output format (%s) as --output is empty", defaultOutputFormat) cmd.Flags().Set("output", defaultOutputFormat) } </ins> printer, _, err := cmdutil.PrinterForCommand(cmd) cmdutil.CheckErr(err)

The panic is bad but I would expect the command to return to me valid outputs I could use and error out. we should also make sure the nil dereference in the resource printer gets handled. Your PR justs masks it.
Btw, I didn't make up the empty string on purpose. It was documented in the output and it was seen as the default setting of . So perhaps someone should also fix the help text so newbies like me don't cause crashes as easily? Thanks.
I'd expect that we validate what's provided to and print the valid options if something unexpected was provided (including blank string). But the way we print flags that expect strings (and defaults to empty) in help can lead to this kind of confusion and other issues (on Windows, for example). A few ideas: pflag could have a "select" type that takes a list of valid options (and validates/throws errors automatically).
Agreed with that the current way of showing flags in the help is not optimal. cc:
I did agree that panic is bad, we should avoid this. The problem is if a user did use an empty output, how to handle this more properly. In PR you commented that we should not override user settings, it seems we should error out directly in this situation. Besides, nil dereference in the resource printer should be handled, will add this.