Error: Failed to run: /usr/lib/lxd/lxd forkstart example1 /var/lib/lxd/containers /var/log/lxd/example1/lxc.conf

I have problem with the lxc when I want to launch a container. Here are the error logs. Could someone lead what could be the problem?
Thanks.

pine64@pine1:/var/lock$ lxc profile show default
config: {}
description: ""
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: br0
    type: nic
  root:
    path: /
    pool: default
    type: disk
name: default
used_by:
- /1.0/containers/example1
pine64@pine1:/var/lock$ lxc launch images:ubuntu/xenial/arm64 example1
Creating example1
Starting example1
Error: Failed to run: /usr/lib/lxd/lxd forkstart example1 /var/lib/lxd/containers /var/log/lxd/example1/lxc.conf: 
Try `lxc info --show-log local:example1` for more info

    pine64@pine1:/var/lock$ lxc info --show-log local:example1
Name: example1
Remote: unix://
Architecture: aarch64
Created: 2018/05/27 06:51 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc 20180527065311.981 ERROR    lxc_container - lxccontainer.c:wait_on_daemonized_start:815 - No such file or directory - Failed to receive the container state
lxc monitor log:
DBUG[05-27|09:51:45] handling                                 ip=@ method=GET url=/1.0/events
DBUG[05-27|09:51:45] New event listener: b9bac84e-95ec-462d-b3c2-9bbb09b9cbff 
DBUG[05-27|09:51:45] Responding to container create 
DBUG[05-27|09:51:45] handling                                 ip=@ method=POST url=/1.0/containers
DBUG[05-27|09:51:45] New task operation: 5e2bd105-b06c-496d-aa42-0e2be8b66090 
DBUG[05-27|09:51:45] Started task operation: 5e2bd105-b06c-496d-aa42-0e2be8b66090 
DBUG[05-27|09:51:45] Connecting to a remote simplestreams server 
DBUG[05-27|09:51:45] handling                                 ip=@ method=GET url=/1.0/operations/5e2bd105-b06c-496d-aa42-0e2be8b66090
DBUG[05-27|09:51:46] handling                                 ip=@ method=GET url=/1.0
DBUG[05-27|09:51:46] handling                                 ip=@ method=GET url=/1.0/containers
DBUG[05-27|09:51:48] Image already exists in the db           image=e19a716f8b1cfaf156dbcbb5cf360d32ccfa146fe64a7b6e7ff298304b42874d
DBUG[05-27|09:51:48] Database error: &errors.errorString{s:"sql: no rows in result set"} 
INFO[05-27|09:51:48] Creating container                       ephemeral=false name=example1
DBUG[05-27|09:51:48] Initializing a DIR driver. 
INFO[05-27|09:51:49] Created container                        ephemeral=false name=example1
DBUG[05-27|09:51:49] Creating DIR storage volume for container "example1" on storage pool "default". 
DBUG[05-27|09:52:06] handling                                 ip=@ method=GET url=/1.0
DBUG[05-27|09:52:06] handling                                 ip=@ method=GET url=/1.0/containers
DBUG[05-27|09:52:06] handling                                 ip=@ method=GET url=/1.0/containers/example1
DBUG[05-27|09:52:48] Shifting root filesystem "/var/lib/lxd/containers/example1/rootfs" for "example1". 
DBUG[05-27|09:53:10] Created DIR storage volume for container "example1" on storage pool "default". 
DBUG[05-27|09:53:10] Success for task operation: 5e2bd105-b06c-496d-aa42-0e2be8b66090 
DBUG[05-27|09:53:10] handling                                 ip=@ method=GET url=/1.0/containers/example1
DBUG[05-27|09:53:10] handling                                 ip=@ method=GET url=/1.0/events
DBUG[05-27|09:53:10] New event listener: 8baa3b1a-bb9d-41ec-ad44-4805361eb7fc 
DBUG[05-27|09:53:10] Disconnected event listener: b9bac84e-95ec-462d-b3c2-9bbb09b9cbff 
DBUG[05-27|09:53:10] handling                                 ip=@ method=PUT url=/1.0/containers/example1/state
DBUG[05-27|09:53:11] Started task operation: fe4dd37a-8599-4c35-8be5-8b174c1eb3b4 
DBUG[05-27|09:53:11] New task operation: fe4dd37a-8599-4c35-8be5-8b174c1eb3b4 
DBUG[05-27|09:53:11] handling                                 ip=@ method=GET url=/1.0/operations/fe4dd37a-8599-4c35-8be5-8b174c1eb3b4
DBUG[05-27|09:53:11] Initializing a DIR driver. 
INFO[05-27|09:53:11] Starting container                       action=start created="2018-05-27 09:51:48 +0300 +03" ephemeral=false name=example1 stateful=false used="1970-01-01 02:00:00 +0200 EET"
DBUG[05-27|09:53:11] handling                                 ip=@ method=GET url=/1.0
DBUG[05-27|09:53:11] handling                                 ip=@ method=GET url=/internal/containers/8/onstart
DBUG[05-27|09:53:11] Initializing a DIR driver. 
DBUG[05-27|09:53:11] Scheduler: container example1 started: re-balancing 
DBUG[05-27|09:53:11] Failure for task operation: fe4dd37a-8599-4c35-8be5-8b174c1eb3b4: Failed to run: /usr/lib/lxd/lxd forkstart example1 /var/lib/lxd/containers /var/log/lxd/example1/lxc.conf:  
EROR[05-27|09:53:11] Failed starting container                action=start created="2018-05-27 09:51:48 +0300 +03" ephemeral=false name=example1 stateful=false used="1970-01-01 02:00:00 +0200 EET"
DBUG[05-27|09:53:12] handling                                 ip=@ method=GET url=/1.0
DBUG[05-27|09:53:12] Disconnected event listener: 8baa3b1a-bb9d-41ec-ad44-4805361eb7fc 
DBUG[05-27|09:53:12] handling                                 ip=@ method=GET url=/1.0/containers

What does lxc config show example1 --expanded show you?
Also the output of lxc info would be useful.

And does the br0 bridge exist on the system?

I have a similar error to the one above… the host and container are both on Debian 9 and LXD is version 3.18. I know this problem get’s fixed when I reboot the host but I’d like to avoid that if possible?

Here are the debugging commands I’ve run so far… please let me know if you’ve any thoughts…

lxc start mycontainer

Leads to

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart mycontainer /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/mycontainer/lxc.conf: 
Try `lxc info --show-log mycontainer` for more info

And in turn lxc info --show-log mycontainer gives

name: mycontainer
Location: none
Remote: unix://
Architecture: x86_64
Created: 2018/12/10 08:06 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc mycontainer 20191014052607.400 ERROR    conf - conf.c:run_buffer:352 - Script exited with status 1
lxc mycontainer 20191014052607.400 ERROR    start - start.c:lxc_init:887 - Failed to run lxc.hook.pre-start for container "mycontainer"
lxc mycontainer 20191014052607.400 ERROR    start - start.c:__lxc_start:1991 - Failed to initialize container "mycontainer"
lxc mycontainer 20191014052607.509 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:864 - No such file or directory - Failed to receive the container state

Then I run

lxc info --show-log mycontainer --expanded

Which returns

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Debian stretch amd64 (20181101_05:24)
  image.os: Debian
  image.release: stretch
  image.serial: "20181101_05:24"
  limits.cpu: "1"
  limits.cpu.priority: "0"
  limits.memory: 1536MB
  volatile.base_image: 998fcf009fc5b08e467cbb0a8e93693f3a4d3b75ca15958df68fabdd560721a9
  volatile.eth0.hwaddr: 00:16:3e:85:b1:b9
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
devices:
  eth0:
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: "2"
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

I also ran lxc monitor when trying to start the container ad this seemed like the pertinent error message but there were a bunch more that came up?

location: none
metadata:
  context: {}
  level: warn
  message: ZFS returned EBUSY while "/var/snap/lxd/common/lxd/storage-pools/2/containers/br-zulip"
    is actually not a mountpoint
timestamp: "2019-10-14T07:37:56.838362693+02:00"
type: logging

Looks like some issues with zfs mounts. You may want to do:

  • grep containers/br-zulip /proc/*/mountinfo

Then look at the pid in one of the results and do:

  • nsenter -t PID -m – umount
    /var/snap/lxd/common/lxd/storage-pools/2/containers/br-zulip

And repeat until the grep returns no match at which point LXD should be able to mount it again.

This particular issue I believe was resolved in upstream zfs a little while back, effectively allowing for the same dataset to be mounted in two different namespaces, but it could be that your version of zfs doesn’t have that fix, or that it’s a different variant of the same issue.

Thank you… that fixed it! I’m running zfs 0.6.5.9-5 on the host… Debian 9…