Containers lost IPv4 access - AppArmor keeps dnsmasq from starting

So, my server and it’s 9 Containers were jugging merrily along until this morning my guys notified me they can’t log in.

lxc list
showed the containers not having any IP addresses anymore.

Further digging showed that there is no dnsmasq running, which would explain that.

So, the only clue as to what the problem may be is this error when I try to restart LXD:
Aug 4 11:11:28 server0 kernel: [ 258.789920] audit: type=1400 audit(1596557488.200:25): apparmor=“STATUS” operation=“profile_replace” info=“same as current profile, skipping” profile=“unconfined” name=“lxd_dnsmasq-lxdbr0_</var/snap/lxd/common/lxd>” pid=4428 comm=“apparmor_parser”
Aug 4 11:11:28 server0 dnsmasq[4429]: cannot read /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw: Permission denied
Aug 4 11:11:28 server0 dnsmasq[4429]: FAILED to start up
Aug 4 11:11:28 server0 kernel: [ 258.800528] audit: type=1400 audit(1596557488.212:26): apparmor=“DENIED” operation=“open” profile=“lxd_dnsmasq-lxdbr0_</var/snap/lxd/common/lxd>” name="/mnt/md1/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw" pid=4429 comm=“dnsmasq” requested_mask=“r” denied_mask=“r” fsuid=0 ouid=0

I am using snap LXD, version 4.4. I have updated the server to Ubuntu 20.04 just this morning.

First off, I cannot find the location of the AppArmor profile of lxd_dnsmasq-lxdbr0_</var/snap/lxd/common/lxd>, so I can’t even move it to complain from enforce.

As such, I have absolutely no idea where to go from here- especially since I do not know what changed to make dnsmasq stop running in the first place…

Please, any suggestions are hugely appreciated!

It looks like your system is using an unsupported setup where /var/snap/lxd or one of its sub-directories is a symlink to /mnt/md1/lxd

This causes the kernel to get very confused when applying path-based security policies and is breaking your setup now that we have added a security layer (apparmor) around dnsmasq.

This isn’t a setup we can really support in general, though changing from a symlink over to a bind-mount should get things working again, at least for now.

Ah! We’ve got a winner.

Okay, so the reason behind this symlink-setup is as follows:
I started the LXD on a small harddrive when the system wasn’t fully configured yet, and then later moved the whole directory to a TB-Raid once I added that into the system because people had already started working on the Containers, so I couldn’t just start from scratch.

So- if you could please point me to the right direction as to how I can move the container storage pool to a different location properly, I guess that would be something for the long run.

Also, how do I do a bind-mount- have actually never heard of such. I will try that.

Thank you so much!
I guess my cowboy-way of doing things really is the issue here. :see_no_evil: :rofl:

Assuming /var/snap/lxd is your symlink, you would do:

  • rm /var/snap/lxd
  • mount -o bind /mnt/md1/lxd /var/snap/lxd

To make this persistent, you will need a line in /etc/fstab for it, something like:

/mnt/md1/lxd /var/snap/lxd none bind defaults 0 0

And because of the mess of mount namespaces in play here, you should do that fstab change and then reboot to clean everything up and get LXD back online.

You could go and manually fix the snap mount namespace with something like:

nsenter --mount=/run/snapd/ns/lxd.mnt mount -o bind /var/lib/snapd/hostfs/mnt/md1/lxd /var/snap/lxd

But I’d strongly recommend the reboot, if only to make sure everything will come back up right the next time the system restarts.

As for doing something like that cleanly in the first place.

You’d normally keep LXD itself on /var/snap/lxd but then make sure its actual data isn’t stored there by:

  • Create a storage pool backed by an external path or disk/partition
    • If using a dir storage backend, you do that with lxc storage create NAME dir source=/some/path
    • Similar instructions for btrfs though btrfs can also use a disk/partition
    • For all the others, point to a disk/partition to use for LXD
  • Configure LXD to place images on the storage pool (storage.images_volume config key)
  • Configure LXD to place backups on the storage pool (storage.backups_volume config key)

Perfect! Thank you so much!!

I will try the bind for now to get my team back to work, and clean up the mess this weekend properly.

Merci beaucoup! Thank you again!