LXD with OVN, Nginx Reverse Proxy working, but

I spent several head scratching months working out how best to meet my goals. Forewarning, I can be dense, if I break out the big crayons when explaining something it’s because I confidently don’t understand it. If I gloss over something, it’s because I don’t even understand how I don’t understand it, lol. Please note that I am self-taught in several IT areas, check out my profile for details if that matters to your responses.

Goals

I want to be able to self host several vlan capable subnets from a single IP; then cluster multiple like devices from one or more IPs … and I eventually landed on LXD due to it’s capabilities as they related to my goals.

Spoiler: thus far everything works, with one peculiarity, hence this post.

Capabilities

  • LXD can host vlan capable subnets from a single IP via OVN;
  • LXD can cluster which is made more attractive with MicroCloud; but waiting on OVN integration.
  • LXD project’s can have their own OVN network; this is a very attractive feature.
  • LXD purports to work with JuJu; I imagine the developers wouldn’t lie about this, right…

@stgraber and team, you all are making a great product.

Controls

  • Host: Ubuntu 22.04, LXD 5.9
  • Containers: Ubuntu 22.04
  • Network: IPV4 only, bridge, uplink pointing back to the bridge (@tomp advised in some random post), ovn subnets. vlan capability is untested.
  • Applications: neovim, nginx, ufw

Implementation

I have OVN setup with the uplink pointing to the lxdbr0 bridge. I have placed a reverse proxy container on the default (lxdbr0) bridge network. For the reverse proxy container, I used the instructions that @simos indicated here excepting that I am self hosting and I am only using Nginx; none of that Apache2 slowness, :stuck_out_tongue: . I have my reverse proxy container attached to two different networks lxdbr0 (bridge) and ovntest (ovn) via separate nics. The ovntest network is associated with project-test. The test website is up and currently running on the internet with SSL as provided by @simos instructions; however due to the nature of OVN, I encountered two different issues:

  1. From within the containers, Nginx consistently failed on nginx -t where-ever I had anything set to <containername>.lxd as provided in the @simos instructions. The work around discovered that where-ever the instructions provided the use of containername.lxd, I replaced that entry with the containers IPV4 address. Which for peace of mind meant manually setting the container’s IP from the host via:

lxc config set <containername> --project=<projectname> ipv4.address=<ip of container>

  1. Once all the IP addresses aligned; I was able to reach my test website container as anticipated, but I then encountered my second OVN related issue. While the web-server and reverse-proxy worked functioned for their intended purpose; I lost all other internet connectivity. Which meant no updates were possible and it that posed security risks, and I wasn’t able to install certbot as the @simos instructions provided. I was able to alleviate this issue by running the following command on the host:

lxc network set uplink dns.nameservers <ip of lxdbr0>

After which, all connectivity returned to the subnets and their associated containers; and everything else proceeded as intended. Life is good.

TL;DR

The one issue that persists is that in order to get the reverse proxy container to connect to all the subnets, I have to run dhclient from within the reverse proxy container after each machine restart. Of course, I can automate a work around from within reverse proxy container, but I didn’t know if there was a configuration setting that I missed. Please advise.

Thanks for your responses,
shimmy

Merry Christmas and Happy Holidays.

1 Like

Please show output of lxc config show <instance> --expanded for the reverse proxy container, as well as ip a and ip r inside that container.

Please also show lxc network show <network> for lxdbr0 and the OVN network.

Thanks

I must apologize for my delayed response, thank-you for taking the time to look into this topic. You guys do make a great product.

TL; DR

Unfortunately, however I cannot do what you ask. I lost that config in a HBA card failure. All said and done, it was a rather catastrophic. As a result, I have spent the last couple of days troubleshooting and now rebuilding the system. I do have an image of the lost data pool, but it’s been resistant to any form of recovery due to lost a partitioning table, damaged sectors, life, etc. I’ll report back if the issue persists after rebuild.

A bit of Humor

What I was able to determine was that the bridge network (lxdbr0) was some how getting mangled without the printed config changing. Running ‘lxc network set uplink dns.nameservers ‘ would resolve the issue, but the config never changed. That seemed so curious to me, it was like there was a ghost config haunting my setup. Looking forward to some paranormal activity; I enthusiastically cued up the X-Files soundtrack on repeat, and with some popcorn I watched one of those “Learn Golang in X hours” videos. As I tried to understand the material, I kept imagining myself as a Ghost Hunter (perhaps I’ll change my moniker). While I was wrestling with the ideas of go modules versus my newb preference for scheduled co-routines… boom! HBA Horror Story! Like Sam and Dean, the unknown monster had come to me and I wasn’t prepared. Hell had broken lose… again!

A bit of Reality

What arose subsequent to the original post and it maybe of some merit for consideration was that the dns handling was continually compromised somewhere between the reverse proxy and the physical nic; and I suspected the bridge. Though this assessment is now overshadowed by the hardware failure.

Like I mentioned in the original post, in every OVN iteration I was able to ping from containers to the internet, but I was not able to download anything; such as the certbot tool as explained above. However, updates to the test http websites were populated as expected. Traffic was getting in but none was getting out.

All the boxes were checked: firewalls, routing, etc. I had diligently followed the instructions on the website, watched @stgraber videos, and I implemented your points from the discussion and thoughts previously explained here, here,and here.

At the time, because I had witnessed the repeated behavior near/during the creation of new OVNs; I had assumed some degree of cause and effect. Typically, I was able to correct this by running the above command.

The last OVN which was right before the HBA issue. Didn’t respond to the command above like the rest. So perhaps, in someway I’ll never fully comprehend, the HBA failure was the contributing factor to the ghost config and the it’s lack response at the end was the last warning?

As mentioned, I am in the process of rebuilding a “like” setup using another HBA controller, and I will build another “like” ovn; the best I can do is report back whether or not the issue persists.

A bit of Wisdom

After all of this, I must comment that ‘self-taught’ means that despite all the setbacks, disappointment, suppositions, and hard learned lessons the person’s passion to endeavor thrives. This diligence is why Edison was able to make the light bulb; without which we would not, as we are, be here and now.

Now, please kindly excuse me while I go off to drown my sorrows…

1 Like