Rbd map fails in nested LXD if LXD installed from snap

LXD
1

I’ve been wanting to try out a ceph backed LXD cluster but have run into the problem that the ceph nodes I’m using were deployed with Juju and already have LXD containers for the Mon and CephFs on them.

Obviously, my next step was to create a (very permissive) container and setup the cluster nested in this new container. After a few modifications I have it working with apt based lxd but using the snap based LXD I can’t get the rbd to map.

If there’s a better way to solve this problem, I’m interested but here’s how I’ve got it working if anyone can help me understand why the snap version is having issues that the apt lxd doesn’t have. I’ve tried the snap lxd w/o nesting (on a different machine) and that works just fine so it’s the combination of nesting and snap that’s just to much for me to untangle.

I’m using this profile for the container that I’m installing snap lxd in.

config:
  raw.lxc: |-
    lxc.cgroup.devices.allow = a
    lxc.cap.drop=
    lxc.apparmor.profile = unconfined
    lxc.mount.auto=
    lxc.mount.auto=sys:rw proc:rw cgroup:rw
  security.nesting: "True"
  security.privileged: "True"
  linux.kernel_modules: rbd
  user.network-config: |
    version: 1
    config:
      - type: physical
        name: eth0
      - type: bridge
        name: br0
        bridge_interfaces:
          - eth0
        subnets:
          - type: dhcp
description: LXD Cluster profile
devices:
  eth0:
    nictype: bridged
    parent: br-enp1s0f0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: lxd

After launching bionic in a container with this profile I

  • Apt remove lxd lxd-client
  • Snap install lxd
  • apt install ceph-common
  • setup /etc/ceph/*.keyring and ceph.conf
  • add udev rules to mknod the new devices
  • run lxd init

/etc/udev/rules.d/99-rbd.rules

# Skip items we don't want to deal with
ACTION=="changed", GOTO="persistent_storage_end"
SUBSYSTEM!="block", GOTO="persistent_storage_end"
KERNEL!="rbd*", GOTO="persistent_storage_end"

# Add partitions
KERNEL=="rbd*", ACTION=="add", RUN+="/bin/sh -c '/bin/echo /bin/mknod $env{DEVNAME} b $env{MAJOR} $env{MINOR} >> /root/udev.log 2>&1'"
KERNEL=="rbd*", ACTION=="add", RUN+="/bin/sh -c '/bin/mknod $env{DEVNAME} b $env{MAJOR} $env{MINOR} >> /root/udev.log 2>&1'"

# Remove partitions
KERNEL=="rbd*", ACTION=="remove", RUN+="/bin/sh -c '/bin/echo rm $env{DEVNAME} >> /root/udev.log 2>&1'"
KERNEL=="rbd*", ACTION=="remove", RUN+="/bin/sh -c 'rm $env{DEVNAME} >> /root/udev.log 2>&1'"

LABEL="persistent_storage_end"

The init completes, but when launching a container the image is never mapped by rbd. With this setup and not switching to the snap everything works as expected.

dmesg during the launch shows:

[ 2854.185360] libceph: mon0 252.2.185.54:6789 session established
[ 2854.185722] libceph: client4572 fsid c0f8c69c-9cf6-11e9-a3b9-00163ecb4cd1
[ 2854.225812] rbd: rbd0: capacity 10000000000 features 0x1

Trying to map it myself

rbd map /dev/rbd0
rbd: sysfs write failed
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (2) No such file or directory
# From dmesg all that prints is
rbd: pool rbd does not exist

I’m not sure why the lxd snap vs apt affect how rbd map runs inside these containers. That’s very strange to me and indicates I’m missing something about how snaps are operating.

What other information would be helpful to capture, or is there a much better way to accomplish this?