LXD - routed config - internet disappears

LXD
1

So I followed this guide (big fan of Simos btw!): How to get LXD containers get IP from the LAN with routed network – Mi blog lah!

It worked great on an Ubuntu 20.04 stock host. Like it just worked like magic. Managed to install and update a few things in the container, all great. However, I had to go out for a few errands so I just left the machine running like that.

When I came back and I wanted to install a few more packages - it could not download them. No more internet. Tried pinging a host, an IP, nothing.

I have no idea what happened, what went wrong. I don’t remember changing anything on the host or the container besides issuing a reboot at some point.

Could the reboot mess it up? How could I make this more “resistant” then?
More info:

  • Host is a Hetzner Cloud VPS.
  • The additional routed IP is a floating IP.
  • Host and guest is Ubuntu 20.04.
  • The host is running Docker, so there are a few network interfaces, but nothing is using this additional IP.
  • This additional IP is NOT present on the host (in /etc/netplan config); I only added it to the routed config as instructed. And that worked…

Let me know if you guys need more info or something!

Update 1: It seems like everything works until I install Docker. As soon as Docker is installed and I do a reboot, the container internet breaks and I cannot bring it back up. But what can cause this? I didn’t even start a Docker container, just installed the base Docker and that already breaks things.

Did this one: curl -sSL https://get.docker.com/ | CHANNEL=stable sh
Once it’s done, things break. I’ll investigate further…

2

Yep it is quite well known that Docker messes with the firewall blocking LXD’s traffic (not just routed, but bridged too) by default.

See Lxd and Docker Firewall Redux - How to deal with FORWARD policy set to drop - #3 by tomp

3

Thank you @tomp !

My goal was to run Mailcow on the “host” and then just dump the other services in LXC “containers” with their own IP. Unfortunately, I had to go with plan B, which is to put Docker on its own specific IP as well

This does require you to use a special config (for nesting).
So I repeat. You only need these special steps if you want to run Docker/mailcow inside. Otherwise, you just need to follow Mr. Simos’s guide for the IP routed thing.

In the end I got my VM working like so:

creation of config:
lxc profile create routed, lxc profile edit routed

creation of the guest:
lxc launch ubuntu nestc1 -c security.nesting=true -c security.privileged=true

modify existing guest:
lxc config set nestc1 security.nesting true
lxc config set nestc1 security.privileged true
lxc restart nestc1

And for the profile:

config:
  raw.lxc: |
    lxc.apparmor.profile = unconfined
    lxc.cgroup.devices.allow = all
    lxc.cap.drop =
    lxc.mount.auto=proc:rw sys:rw
  user.network-config: |
    version: 2
    ethernets:
        eth0:
            addresses:
            - 192.168.1.200/32
            nameservers:
                addresses:
                - 8.8.8.8
                search: []
            routes:
            -   to: 0.0.0.0/0
                via: 169.254.0.1
                on-link: true
description: ""
devices:
  eth0:
    ipv4.address: 192.168.1.200
    nictype: routed
    parent: eth0
    type: nic
name: routed1
used_by: []
  • These “unsecure” changes were applied so Mailcow/Docker would run inside.
  • I modified the “cgroup.devices.allow” bit to “all” instead of specific bits, because I do NOT want my “guest” to break on me during a random update. I run verified stuff inside, I am fairly certain I’ll be fine security wise.
  • For other “guests”, you do not need the “raw.lxc” bit, or the special “creation” bit at the beginning.

Fingers crossed my things won’t break.

Unfortunately, Hetzner raised their IP prices like crazy (300-400% increase), so the whole plan to save by moving my stack to “Cloud” kinda failed. I still end up spending less, but the savings are way fewer thanks to the price increase now. It is what it is. :frowning: