LXD 3.18 - install from source - unable to start containers

LXD
#1

Hello,
I have been stuck on this issue for quite some time, and the problem is I am not sure how someone can reproduce this issue. I have installed the same LXD version on my other machines (which are all Centos 8 but different kernels; all 5.x above) from source and they work alright. This particular one messes up when starting the container. I imported my image and was able to create a container.

Also, in this install, no group named lxd was created (whereas in other installs, it was). Anyway, on trying to start my container, I get this error:
Error: Failed to start container: Failed to run: /home/alpha/go/bin/lxd forkstart droid0 /var/lib/lxd/containers /var/log/lxd/droid0/lxc.conf

Logs:

Name: droid0
Location: none
Remote: unix://
Architecture: x86_64
Created: 2020/07/31 21:27 UTC
Status: Stopped
Type: container
Profiles: default

Log:

    lxc droid0 20200731214615.714 INFO     lxccontainer - lxccontainer.c:do_lxcapi_start:971 - Set process title to [lxc monitor] /var/lib/lxd/containers droid0
    lxc droid0 20200731214615.714 INFO     start - start.c:lxc_check_inherited:311 - Closed inherited fd 4
    lxc droid0 20200731214615.714 INFO     start - start.c:lxc_check_inherited:311 - Closed inherited fd 5
    lxc droid0 20200731214615.714 INFO     start - start.c:lxc_check_inherited:311 - Closed inherited fd 6
    lxc droid0 20200731214615.715 INFO     lsm - lsm/lsm.c:lsm_init:50 - LSM security driver SELinux
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "reject_force_umount  # comment this to allow umount -f;  not recommended"
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for reject_force_umount action 0(kill)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for reject_force_umount action 0(kill)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for reject_force_umount action 0(kill)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:do_resolve_add_rule:505 - Set seccomp rule to reject force umounts
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for reject_force_umount action 0(kill)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "[all]"
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "kexec_load errno 38"
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for kexec_load action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for kexec_load action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for kexec_load action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for kexec_load action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "open_by_handle_at errno 38"
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for open_by_handle_at action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for open_by_handle_at action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for open_by_handle_at action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for open_by_handle_at action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "init_module errno 38"
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for init_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for init_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for init_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for init_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "finit_module errno 38"
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for finit_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for finit_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for finit_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for finit_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:759 - Processing "delete_module errno 38"
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:937 - Added native rule for arch 0 for delete_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:946 - Added compat rule for arch 1073741827 for delete_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:956 - Added compat rule for arch 1073741886 for delete_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:966 - Added native rule for arch -1073741762 for delete_module action 327718(errno)
    lxc droid0 20200731214615.715 INFO     seccomp - seccomp.c:parse_config_v2:970 - Merging compat seccomp contexts into main context
    lxc droid0 20200731214615.715 INFO     conf - conf.c:run_script_argv:374 - Executing script "/proc/9671/exe callhook /var/lib/lxd 8 start" for container "droid0"
    lxc droid0 20200731214615.733 DEBUG    conf - conf.c:run_buffer:340 - Script exec /proc/9671/exe callhook /var/lib/lxd 8 start produced output: Error: Failed to run: apparmor_parser --version:

    lxc droid0 20200731214615.734 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 1
    lxc droid0 20200731214615.734 ERROR    start - start.c:lxc_init:881 - Failed to run lxc.hook.pre-start for container "droid0"
    lxc droid0 20200731214615.734 ERROR    start - start.c:__lxc_start:1971 - Failed to initialize container "droid0"
    lxc droid0 20200731214615.734 INFO     conf - conf.c:run_script_argv:374 - Executing script "/home/alpha/go/bin/lxd callhook /var/lib/lxd 8 stopns" for container "droid0"
    lxc droid0 20200731214645.749 DEBUG    conf - conf.c:run_buffer:340 - Script exec /home/alpha/go/bin/lxd callhook /var/lib/lxd 8 stopns produced output: Error: Hook didn't finish within 30s

    lxc droid0 20200731214645.749 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 1
    lxc droid0 20200731214645.749 ERROR    start - start.c:lxc_fini:1007 - Failed to run "lxc.hook.stop" hook
    lxc droid0 20200731214645.752 DEBUG    lxccontainer - lxccontainer.c:wait_on_daemonized_start:839 - First child 10425 exited
    lxc droid0 20200731214645.752 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - No such file or directory - Failed to receive the container state
    lxc 20200731214645.753 WARN     commands - commands.c:lxc_cmd_rsp_recv:135 - Connection reset by peer - Failed to receive response for command "get_init_pid"
    lxc 20200731214645.753 WARN     commands - commands.c:lxc_cmd_rsp_recv:135 - Connection reset by peer - Failed to receive response for command "get_state"

Any help @stgraber? (TIA)
~

#2

Seems to be complaining about apparmor_parser --version

#3

But its CentOS (and I am using SELinux), and this never came up in the other CentOS instances of mine.
How can I try and fix this issue?

#4

Maybe those other systems don’t have the apparmor_parser command present?

Any chance you can also try a recent version of LXD, that would make it a bit easier to track down how you could get into that particular code path in the first place.

#5

This system does not have that command either :open_mouth:
root# app <and I press tab>
applygnupgdefaults appstream-compose appstream-util
I can try using another version, then I would have to completely delete LXD from my system (as if it was never installed before). Can you help me please? :slight_smile:

In the logs above, I see this-> INFO: lsm - lsm/lsm.c:lsm_init:50 - LSM security driver SELinux

Also, my kernel config: CONFIG_SECURITY_APPARMOR is not set

TIA
~

#6

What LXD release is that?

We just fixed a related issued affecting 4.4: https://github.com/lxc/lxd/pull/7718

#7

The one I am using is 3.18. Why would this issue not happen on my other instances is another problem :slight_smile:

They all are Centos 8 and are using LXD 3.18

~