LXD 3 not launching on ZFS dataset

Graham · June 11, 2018, 10:52pm

I’m really stuck here and hoping someone can help me out as I’m new to LXD and probably missing something blindingly obvious!

I’ve managed to build myself a shiny new Ubuntu 18.04-LTS system using ZFS on LUKS for the root filesystem but when I try to ‘lxc launch’ something it just downloads the image and then fails.

I configured LXD INIT to use a ZFS dataset (rather than a new pool) called rpool/lxd, which it duly created:

# zfs list

NAME                                                                                USED  AVAIL  REFER  MOUNTPOINT
rpool                                                                              2.62G  7.65T   140K  /
rpool/ROOT                                                                         2.05G  7.65T   140K  none
rpool/ROOT/ubuntu                                                                  2.05G  2.95G  1.81G  /
rpool/home                                                                         18.7M  4.98G  18.4M  /home
rpool/home/root                                                                     273K  4.98G   273K  /root
rpool/lxd                                                                           441M   500G   140K  none
rpool/lxd/containers                                                               11.7M   500G   140K  none
rpool/lxd/custom                                                                    140K   500G   140K  none
rpool/lxd/deleted                                                                   140K   500G   140K  none
rpool/lxd/images                                                                    429M   500G   140K  none
rpool/lxd/snapshots                                                                 140K   500G   140K  none
rpool/var                                                                           117M  4.89G   140K  /var
rpool/var/cache                                                                     105M  4.89G   105M  /var/cache
rpool/var/log                                                                      11.3M  4.89G  11.3M  legacy
rpool/var/nfs                                                                       140K  4.89G   140K  /var/lib/nfs
rpool/var/spool                                                                     198K  4.89G   198K  /var/spool
rpool/var/tmp                                                                       174K  4.89G   174K  legacy

And then I try to launch a new container:

# lxc launch ubuntu:xenial MyFirstContainer

Creating MyFirstContainer
Starting MyFirstContainer
Error: Failed to run: /usr/lib/lxd/lxd forkstart MyFirstContainer /var/lib/lxd/containers /var/log/lxd/MyFirstContainer/lxc.conf:
Try `lxc info --show-log local:MyFirstContainer` for more info

Using the suggested lxc info command I get:

Name: MyFirstContainer
Remote: unix://
Architecture: x86_64
Created: 2018/06/11 22:34 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc 20180611223503.982 ERROR    lxc_lsm - lsm/lsm.c:lsm_process_label_set_at:167 - No such file or directory - Failed to set AppArmor label "lxc-container-default-cgns"
lxc 20180611223503.982 ERROR    lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:243 - No such file or directory - Failed to change apparmor profile to lxc-container-default-cgns
lxc 20180611223503.982 ERROR    lxc_sync - sync.c:__sync_wait:57 - An error occurred in another process (expected sequence number 5)
lxc 20180611223504.219 ERROR    lxc_container - lxccontainer.c:wait_on_daemonized_start:824 - Received container state "ABORTING" instead of "RUNNING"
lxc 20180611223504.219 ERROR    lxc_start - start.c:__lxc_start:1866 - Failed to spawn container "MyFirstContainer"
lxc 20180611223504.298 WARN     lxc_commands - commands.c:lxc_cmd_rsp_recv:130 - Connection reset by peer - Failed to receive response for command "get_cgroup"

Whilst I’m not new to Linux or ZFS, I nearly always use RHEL/CentOS and so I am new to AppArmor and LXD; is this is an AppArmor issue I’m seeing here or something else?

It looks like the launch command is creating the required ZFS datasets but since I’m new to this I don’t know if this looks the way it should!

# zfs list | grep lxd

rpool/lxd                                                                           441M   500G   140K  none
rpool/lxd/containers                                                               11.7M   500G   140K  none
rpool/lxd/containers/MyFirstContainer                                              11.5M   500G   429M  /var/lib/lxd/storage-pools/lxd/containers/MyFirstContainer
rpool/lxd/custom                                                                    140K   500G   140K  none
rpool/lxd/deleted                                                                   140K   500G   140K  none
rpool/lxd/images                                                                    429M   500G   140K  none
rpool/lxd/images/08bbf441bb737097586e9f313b239cecbba96222e58457881b3718c45c17e074   428M   500G   428M  none
rpool/lxd/snapshots                                                                 140K   500G   140K  none

Can anyone provide me with any pointers where to look? I’m so near, yet so far!

Thanks.

dabeegmon · June 11, 2018, 11:47pm

Just another user here so I might also be way out there (have been in that place quite a few times!!).

Did you actually install an image?

https://stgraber.org/2016/03/19/lxd-2-0-your-first-lxd-container-312/

This page is explicitly for lxd 2.xx but I think the first part (lxc launch . . . .) is where I would be looking.
If this doesn’t help - - - my apologies and one of the dev team should be along shortly!

simos · June 12, 2018, 9:58am

Hi!
The very first error is not being able to set an AppArmor label. Your setup looks fine, with the exception of LUKS which might or might not be the issue.

Can you try to setup again without LUKS?

Do you run the LXD snap or the Deb package?

Graham · June 12, 2018, 12:47pm

Thanks for your input dabeegmon, I checked out your link and it looks like I’m trying to start things the correct way and I can see both an image and a container listed on my setup, but for some reason the container fails to start.

# lxc image list

+-------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |                 DESCRIPTION                 |  ARCH  |   SIZE   |          UPLOAD DATE          |
+-------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+
|       | 08bbf441bb73 | no     | ubuntu 16.04 LTS amd64 (release) (20180522) | x86_64 | 157.43MB | Jun 11, 2018 at 10:34pm (UTC) |
+-------+--------------+--------+---------------------------------------------+--------+----------+-------------------------------+

# lxc list

+------------------+---------+------+------+------------+-----------+
|       NAME       |  STATE  | IPV4 | IPV6 |    TYPE    | SNAPSHOTS |
+------------------+---------+------+------+------------+-----------+
| MyFirstContainer | STOPPED |      |      | PERSISTENT | 0         |
+------------------+---------+------+------+------------+-----------+

Graham · June 12, 2018, 1:01pm

Hi Simos. thanks for your response.

I’m using the stock LXD (3.0.0-0ubuntu4) that installs via APT on Ubuntu 18.04-LTS.

Rebuilding this machine is a big job which I’d rather avoid if possible. Since LUKS/DM-Crypt sits under ZFS it should be invisible, is it possible for AppArmor to have interactions with dm-crypt in this way?

If there’s really nowhere else to look then I’ll try a rebuild, but I’d really rather avoid it if possible :-s

Is there a way to disable AppArmor, kind of the equivalent to setting SELinux to Permissive mode on RHEL?

Thanks again for your help, it is appreciated.

Graham.

Graham · June 12, 2018, 1:09pm

As is so often the case with these things, just describing the problem and asking some questions in reply to the responses I got has lindirectly lead me to the solution!

After asking the question if there is a way to disable AppArmor I googled that topic to find out how to do it and shortly afterwards realised that AppArmor wasn’t even installed on my system!

As part of the process of building Ubuntu 18.04 with root on ZFS on dm-crypt you end up installing a very minimal system, which is what I wanted, but it seems it’s so minimal that it doesn’t even include AppArmor, so I installed AppArmor and now I can launch containers!

However, this does lead me to the thought that perhaps LXD should have AppArmor as a dependency and that APT should install AppArmor along with LXD.

Thank you both for your help!

Graham

P.S. The system I’m building is for my home lab and will be an Ubuntu 18.04-LTS with root on encrypted ZFS (via dm-crypt), running KVM, LXD and Docker (in LXD) - probably with Kubernetes and Rancher or Tungsten Fabric. If anyone is interested in building a similar setup and would like me to write it up when I’m done then please let me know

simos · June 12, 2018, 1:25pm

Thanks for working this out.
It’s important that you wrote the error messages which means that if a user encounters the same issue and they google the exact same error message, they will be directed here.

AppArmor is both a kernel module and a set of libraries. Can you check your /var/log/apt/ to see which packages were installed from the AppArmor libraries? That is, which package did you install that managed to pull in all the rest of the required packages to make LXD work.

In terms of the usability of LXD, it would be great if there was a check somewhere to inform on the missing AppArmor libraries. Normally, there are informative messages when something is missing. This was a bit of a corner case.

It would be great if you could write a tutorial on setting up all these, I presume starting from the mini.iso, doing encrypted ZFS and setting up the rest. I have not done such a setup so I would gladly use a tutorial to go through it.

Graham · June 12, 2018, 1:54pm

As far as AppArmor goes I just did “apt install apparmor” from 18.04 repos. The log shows this:

Start-Date: 2018-06-12  13:57:05
Commandline: apt install apparmor
Install: apparmor:amd64 (2.12-4ubuntu5)
End-Date: 2018-06-12  13:57:10

I suppose mine is a bit of a corner case, but only because the installer doesn’t yet support root ZFS on dm-crypt (catch up Canonical !!) otherwise I think a lot more people would be doing it.

LXD clearly has a dependency on AppArmor and so I’d expect it to be listed in the apt package spec so that apt will install AppArmor for LXD if it’s missing. Anyhow, problem solved and now I can start exploring LXD and find some more hurdles to jump

I actually built my system starting with a LiveCD Desktop rather than the mini.iso, which meant I could wipe all my drives and start with a completely bare system. When I get time to write it up - probably not too soon - where’s the best place to post it? Can I write it directly on this board or is there a better wiki somewhere I can provide a link to?

Graham

simos · June 12, 2018, 4:32pm

If you use github, you can create a page on myusername.github.io.
Alternatively, you can make a new post on this discussion forum and write the tutorial. This forum supports markdown, so you can create lists, tables, add images, etc.

I’ll keep a mental note to check for apparmor when I next install 18.04.

dabeegmon · June 12, 2018, 9:38pm

Sorry - - - I landed up wasting your time - - - not what I wanted to do.
I didn’t see the image (wasn’t like what I’m used to seeing).

Hope you didn’t take too much time at my rabbit trail!

Regards

Graham · June 12, 2018, 10:47pm

You certainly didn’t waste my time and I’m most appreciative of the time you took to look at my post and to offer help!

Best wishes,

Graham