Nested LXD container with debian as outer container - any change?

I know that this didn’t work (or at least was very tricky to get working) and you had to have ubuntu as the outer container. Has this changed at all since lxd 4.0 / buster came out?

If anyone has a link to a short explanation as to why it doesn’t work, I’d be interested in that too. Presumably it’s not kernel related since both outer containers would be running same kernel as the host.

TIA

Hi!

Can you show us a test run? That is,

  1. mention what’s running on the host (example, Ubuntu 20.04, with LXD 4.4 from snap package).
  2. show the launch of the outer container, with security.nesting=true.
  3. [tricky part] install LXD in the outer Debian container, which LXD package?
  4. launch a container from within the outer container, show any error messages.

Host is a debian buster system and trying to nest containers fails at the first step with an outer debian container but works fine with an outer ubuntu container.

Host

$ uname -a
Linux uriel 4.19.0-10-amd64 #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linux

$ cat /etc/debian_version 
10.5

$ snap list
Name    Version   Rev    Tracking       Publisher   Notes
core    16-2.45   9289   latest/stable  canonical✓  core
core18  20200427  1754   latest/stable  canonical✓  base
lxd     4.0.1     15451  4.0/candidate  canonical✓  -

Debian outer container

$ lxc config show debian-outer
architecture: x86_64
config:
  raw.lxc: lxc.apparmor.profile=unconfined
  security.nesting: "true"
  security.privileged: "true"
  volatile.base_image: cd0c342eff8d4821995df3dd3ab5d5da6909a3fa3dd472e09b2c95b54204b9dc
  volatile.eth0.host_name: vethe21f893b
  volatile.eth0.hwaddr: 00:16:3e:31:92:3f
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: RUNNING
devices: {}
ephemeral: false
profiles:
- onmain
stateful: false
description: ""


root@debian-outer:~# apt-get install snapd apparmor-profiles-extra apparmor-utils 
...
The following additional packages will be installed:
  apparmor ca-certificates libudev1 openssl python3-apparmor python3-libapparmor squashfs-tools udev


root@debian-outer:~# snap install hello-world
error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
       /tmp/sanity-mountpoint-385681172: mount failed: Operation not permitted.

Ubuntu outer container

$ lxc config show ubuntu-outer
architecture: x86_64
config:
  image.architecture: amd64
  image.description: Ubuntu focal amd64 (20200806_07:42)
  image.os: Ubuntu
  image.release: focal
  image.serial: "20200806_07:42"
  image.type: squashfs
  security.nesting: "true"
  volatile.apply_template: copy
  volatile.base_image: ec0fc13a22566f8523b47c7a0d254ad599e3e34879f3d706aec0b0bef025579f
  volatile.eth0.hwaddr: 00:16:3e:6b:19:ed
  volatile.idmap.base: "0"
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
devices: {}
ephemeral: false
profiles:
- default
stateful: false
description: ""


root@ubuntu-outer:~# snap install hello-world
hello-world 6.4 from Canonical✓ installed

# have actually already done snap install lxd...

root@ubuntu-outer:~# lxc launch images:debian/buster
Creating the instance
Instance name is: vast-owl                  
Starting vast-owl

Thanks for the nice presentation.

A suggestion first; I notice that for LXD on the host you are tracking 4.0/candidate'. Here you can either switch to 4.0/stableor let it trackstable` (which means it will upgrade to LXD 4.4 (currently).

The exact issue is that in a Debian container, snapd cannot mount using squashfs, while in an Ubuntu container it is fine.
According to https://github.com/lxc/lxc/issues/1854, it is not possible to mount a squashfs but rather you can use squashfuse instead. Therefore, install squashfuse in Debian and try again.

Thanks for the reminder re: “candidate” channel Simos - I set it to that to pick up a bugfix a while ago when snap “helpfully” upgraded lxd behind my back and broke all my containers.

I had already tried the squashfuse package, in case that provided what was needed. All I got was a slightly different error message I’m afraid.

root@debian-outer:~# dpkg-query --list 'squash*' | grep ii
ii  squashfs-tools 1:4.3-12     amd64        Tool to create and append to squashfs filesystems
ii  squashfuse     0.1.103-1    amd64        FUSE filesystem to mount squashfs archives

root@debian-outer:~# snap install hello-world
error: system does not fully support snapd: cannot mount squashfs image using "fuse.squashfuse":
       mount: /tmp/sanity-mountpoint-939393695: wrong fs type, bad option, bad superblock on
       /tmp/sanity-squashfs-352129600, missing codepage or helper program, or other error.

I managed to capture a copy of the temporary squashfile it tried to open and manually mounted it in /tmp just to check

root@debian-outer:/tmp# squashfuse xsquash mountpoint

root@debian-outer:/tmp# ls mountpoint/
canary.txt

Of course, there doesn’t seem to be a debug/verbose option to snap.

However, I can get the same error message as it gives with this:

root@debian-outer:/tmp# mount -t fuse.squashfuse xsquash mountpoint/

mount: /tmp/mountpoint: wrong fs type, bad option, bad superblock on /tmp/xsquash, missing codepage or helper program, or other error.

Where it is getting that filesystem type from is a mystery to me though. I don’t see any reason to think it is supplied by the squash/fuse packages in debian.