So, I’ve been beating my head against this problem for days now and don’t appear to be any closer to a solution. The issue is this:
I have an LXD cluster in Google’s Cloud, GCP. I want to run services on it that can be accessed from GKE and various other points. However, being that this is a cluster mere port forwarding is not a good solution as that would immediately break if a container is moved from the host it was initially on. My initial thought was to get the containers their own IPs that can be reached directly from outside the cluster. So far, this has completely failed. Next, I thought about a service mesh type of thing. However, I cannot find any documentation for doing one that doesn’t immediately require k8s or similar things, so that does me no good either. Were all the traffic HTTP based, I’d probably just use Caddy as a front end and be done with it, but it isn’t, so I can’t.
So, what are my options that anyone knows of? There was a post on here a while back that claims to have solved it, but it provided so little context and information it wasn’t useful. Does anyone have any ideas of options on how to solve this?
It’s not an aspect of GCP that I’m super familiar with but you may be able to route a static address to the cloud instance running your LXD instance, then route that into the LXD instance. When moving to another cloud instance, you can then reconfigure the VPC to route the address to the new host.
Looking at their docs, they seem to have a cloud router which allows for BGP peering. If that works the way I’d hope it does, it should allow for the creation of a VPC using dynamic routing which can then use our current work on BGP advertisements of addresses and routes to dynamically move those routes around.
Hmmm… that might work, but I have the feeling the Powers That Be might not be thrilled with it. Any thoughts on a VXLAN gateway in on the GKE side and integrating that into the Ubuntu FAN?
Just had a quick try to see if this would work and annoyingly Google doesn’t let you BGP peer between one of their cloud routers and instances on the VPC. The BGP peering feature is exclusively restricted to direct connections or VPNs.
I even tried to establish a VPN between the VPC and an instance on it (as ridiculous as it sounds) but they also prevent that by detecting that the peer address is one of theirs…
So as cool as it would have been to get this all to work that way, it doesn’t look like Google wants you to do it… I did confirm that you can effectively add static routes to the VPC which points to a particular instance that then forwards it to the container.
This works fine to have an instance on the VPC reach a container inside of another instance and I suspect would work fine for a Kubernetes pod too. The main downsides to this are that you need to either manually update those routes when moving the LXD container or would need to have an API script that does it for you. Additionally, Google’s NAT doesn’t apply to those routed subnets, so you’d need to double NAT outgoing traffic to have it reach the outside world.
If you control the host VM on the GKE side and it runs Ubuntu (a requirement for the Fan as that’s not an upstream feature), then you should be able to manually configure a Fan interface which will then let you route to instances on any fan bridge.
Note however that because of how the Fan works, moving a container between two cloud instances will cause the address to change.
If you control the host VM on the GKE side and it runs Ubuntu (a requirement for the Fan as that’s not an upstream feature), then you should be able to manually configure a Fan interface which will then let you route to instances on any fan bridge.
I’m not sure what the status of that, trying to find out now. If we don’t, is there any reason we couldn’t create one specifically to act as a FAN gateway?
Note however that because of how the Fan works, moving a container between two cloud instances will cause the address to change.
True, but we can handle this via DNS and such so not an issue
I mean, in theory you should be able to go in the console and put your entire FAN /8 subnet as a route for the VPC with any of your instances as the next-hop.
A k8s pod would then hit the VPC gateway which would forward onto that one instance which then forwards onto the FAN. Worth noting that you need to configure that target instance to have “Allow forwarding” on its network interface otherwise it won’t work.
I have quite a lot of experience on this topic. Right now i’m using basic port forwarding method. I faced the same problem as you where I didn’t want to be dependent on port forwarding. I’ve come to the conclusion that to not be locked down via a port I will need some intelligent routing in my cluster which sits between the external load balancer and internal network.
This router is a piece of software i’ve been working towards, it will be open sourced and free for everyone to use. Essentially it will be smart enough to be aware which domain maps to which container inside the cluster, and maps the traffic intelligently. IT would essentially work like this
all domain → 1 port let’s say 49150 → intelligent router
The router then has some configuration which tells it which domain maps to which app, and does it’s job. The router will need to be aware of LXD because it needs to be able to dynamically fetch containers and map services intelligently with least configuration as possible.
I believe there are existing software that does this however from my research there are a lot of complex configuration that has high maintenance cost which has lead me to my current solution, which is to use named ports on the Google load balancer to map the traffic to the host.
So for example 1 of my app would reserve a certain port let’s say
49182 → maps to app 1
49183 → maps to app 2
And from the google load balancer i map the port to a back-end service which I then map to the domain. This way I can use the load balancer’s full feature of health checks / automatic SSL certificate renewal etc… It’s not ideal, but in the short run it works and comes with a lot of things out of the box.
It’s usually recommended to be an unused /8 (like 240.0.0.0/8), in theory the FAN could deal with a /16 so long as you don’t need more than the last 8 bit of your host’s addresses encoded in the IP and don’t need more than 8 bit to encode all the instances on the host, but I’m not sure that LXD supports that.
Along these lines, I’m wondering. What’s the recommended way for getting into a LXD cluster? In k8s, you’d likely use an ingress manager or something like that. While you can use port forwards I can see that getting complicated to say the least. Surely there’s a better way, no?
So, we tried using an Alias network (but without routing on the VMs) and so far, that doesn’t work. The problem is that the alias network is attached to the host and I haven’t yet figured out how to either get that unattached from the host or to get the host to recognize it needs to do something with those packets instead of just answering itself
