Hi
I recently suffered a system disk failure which took out my lxd tree but left all my container O/S files safe and sound on a separate zfs pool. I tried to recover them using ‘lxd recover’ but following issues trying to get that to work I decided to copy out my containers’ o/s files and create import sets so as to ‘lxc import’ them into the newly restored machine. The import appears to have worked. lxc import created two images without complaint. Then I tried to launch them to create the containers and both images failed to launch a container. Lxc appears to have created the containers but failed on the ‘start container’ step. The containers can be listed but when I try to start them I get the same failure message.
Create the Image:
root#> lxc image import ABC-meta.tar.gz ABC-root.tar.gz --alias ABC
Image imported with fingerprint: b43c9ddbe7d49ca118638c4247a87af5db4e824e09e2f2cdff7d9d519ded0534
Launch the container:
root#> lxc launch ABC ABC-Container
Creating ABC-Container
The instance you are starting doesn't have any network attached to it.
To create a new network, use: lxc network create
To attach a network to an instance, use: lxc network attach
Starting ABC-Container
Error: Failed to run: /snap/lxd/current/bin/lxd forkstart ABC-Container /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/ABC-Container/lxc.conf:
Try `lxc info --show-log local:ABC-Container` for more info
View the log:
root#> lxc info --show-log local:ABC-Container
Name: ABC-Container
Location: none
Remote: unix://
Architecture: x86_64
Created: 2023/03/04 00:26 UTC
Status: Stopped
Type: container
Profiles: default
Log:
lxc ABC-Container 20230304002908.400 WARN conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc ABC-Container 20230304002908.400 WARN conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc ABC-Container 20230304002908.401 WARN conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc ABC-Container 20230304002908.401 WARN conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc ABC-Container 20230304002908.402 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1252 - No such file or directory - Failed to fchownat(40, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )
lxc ABC-Container 20230304002908.467 ERROR start - start.c:start:2164 - No such file or directory - Failed to exec "/sbin/init"
lxc ABC-Container 20230304002908.467 ERROR sync - sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 7)
lxc ABC-Container 20230304002908.468 ERROR lxccontainer - lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING"
lxc ABC-Container 20230304002908.469 ERROR start - start.c:__lxc_start:2074 - Failed to spawn container "ABC-Container"
lxc ABC-Container 20230304002908.469 WARN start - start.c:lxc_abort:1039 - No such process - Failed to send SIGKILL via pidfd 41 for process 258252
lxc ABC-Container 20230304002908.550 WARN conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc ABC-Container 20230304002908.550 WARN conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc 20230304002908.588 ERROR af_unix - af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20230304002908.589 ERROR commands - commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"
View the lxd log:
root#> vi /var/snap/lxd/common/lxd/logs/lxd.log
t=2023-03-04T01:50:26+0000 lvl=info msg="LXD is starting" mode=normal path=/var/snap/lxd/common/lxd version=4.0.8
t=2023-03-04T01:50:26+0000 lvl=info msg="Kernel uid/gid map:"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - u 0 0 4294967295"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - g 0 0 4294967295"
t=2023-03-04T01:50:26+0000 lvl=info msg="Configured LXD uid/gid map:"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - u 0 1000000 1000000000"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - g 0 1000000 1000000000"
t=2023-03-04T01:50:26+0000 lvl=info msg="Kernel features:"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - closing multiple file descriptors efficiently: no"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - netnsid-based network retrieval: yes"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - pidfds: yes"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - core scheduling: no"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - uevent injection: yes"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - seccomp listener: yes"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - seccomp listener continue syscalls: yes"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - seccomp listener add file descriptors: no"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - attach to namespaces via pidfds: no"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - safe native terminal allocation : yes"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - unprivileged file capabilities: yes"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - cgroup layout: hybrid"
t=2023-03-04T01:50:26+0000 lvl=warn msg=" - Couldn't find the CGroup blkio.weight, disk priority will be ignored"
t=2023-03-04T01:50:26+0000 lvl=warn msg=" - Couldn't find the CGroup memory swap accounting, swap limits will be ignored"
t=2023-03-04T01:50:26+0000 lvl=info msg=" - shiftfs support: disabled"
t=2023-03-04T01:50:26+0000 lvl=info msg="Initializing local database"
t=2023-03-04T01:50:26+0000 lvl=info msg="Set client certificate to server certificate" fingerprint=c916925cb1195da8d464956e0499d5de1ca41a846890598722f9fee078a1d634
t=2023-03-04T01:50:26+0000 lvl=info msg="Starting database node" address=1 id=1 role=voter
t=2023-03-04T01:50:27+0000 lvl=info msg="Starting /dev/lxd handler:"
t=2023-03-04T01:50:27+0000 lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock
t=2023-03-04T01:50:27+0000 lvl=info msg="REST API daemon:"
t=2023-03-04T01:50:27+0000 lvl=info msg=" - binding Unix socket" inherited=true socket=/var/snap/lxd/common/lxd/unix.socket
t=2023-03-04T01:50:27+0000 lvl=info msg="Initializing global database"
t=2023-03-04T01:50:27+0000 lvl=info msg="Connecting to global database"
t=2023-03-04T01:50:27+0000 lvl=info msg="Connected to global database"
t=2023-03-04T01:50:27+0000 lvl=info msg="Initialized global database"
t=2023-03-04T01:50:27+0000 lvl=info msg="Firewall loaded driver" driver=xtables
t=2023-03-04T01:50:27+0000 lvl=info msg="Initializing storage pools"
t=2023-03-04T01:50:27+0000 lvl=info msg="Initializing daemon storage mounts"
t=2023-03-04T01:50:27+0000 lvl=info msg="Loading daemon configuration"
t=2023-03-04T01:50:27+0000 lvl=info msg="Initializing networks"
t=2023-03-04T01:50:27+0000 lvl=info msg="Pruning leftover image files"
t=2023-03-04T01:50:27+0000 lvl=info msg="Done pruning leftover image files"
t=2023-03-04T01:50:27+0000 lvl=info msg="Starting device monitor"
t=2023-03-04T01:50:27+0000 lvl=warn msg="Failed to initialize fanotify, falling back on fsnotify" err="Failed to initialize fanotify: invalid argument"
t=2023-03-04T01:50:27+0000 lvl=info msg="Started seccomp handler" path=/var/snap/lxd/common/lxd/seccomp.socket
t=2023-03-04T01:50:27+0000 lvl=info msg="Pruning expired images"
t=2023-03-04T01:50:27+0000 lvl=info msg="Done pruning expired images"
t=2023-03-04T01:50:27+0000 lvl=info msg="Pruning expired instance backups"
t=2023-03-04T01:50:27+0000 lvl=info msg="Done pruning expired instance backups"
t=2023-03-04T01:50:27+0000 lvl=info msg="Expiring log files"
t=2023-03-04T01:50:27+0000 lvl=info msg="Updating instance types"
t=2023-03-04T01:50:27+0000 lvl=info msg="Done updating instance types"
t=2023-03-04T01:50:27+0000 lvl=info msg="Done expiring log files"
t=2023-03-04T01:50:27+0000 lvl=info msg="Daemon started"
t=2023-03-04T01:50:43+0000 lvl=info msg="Starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T00:48:59+0000
t=2023-03-04T01:50:49+0000 lvl=eror msg="Failed starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T00:48:59+0000
t=2023-03-04T02:34:35+0000 lvl=info msg="Starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T01:50:44+0000
t=2023-03-04T02:34:41+0000 lvl=eror msg="Failed starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T01:50:44+0000
t=2023-03-04T02:50:27+0000 lvl=info msg="Pruning expired instance backups"
t=2023-03-04T02:50:27+0000 lvl=info msg="Done pruning expired instance backups"
t=2023-03-04T03:06:09+0000 lvl=info msg="Starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T02:34:36+0000
t=2023-03-04T03:06:15+0000 lvl=eror msg="Failed starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T02:34:36+0000
t=2023-03-04T03:13:39+0000 lvl=info msg="Starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T03:06:10+0000
t=2023-03-04T03:13:44+0000 lvl=eror msg="Failed starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T03:06:10+0000
t=2023-03-04T03:17:26+0000 lvl=info msg="Starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T03:13:39+0000
t=2023-03-04T03:17:31+0000 lvl=eror msg="Failed starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T03:13:39+0000
t=2023-03-04T03:25:03+0000 lvl=info msg="Syncing image to members started" fingerprint=313f9ad6d74524b0d13f4ed9fbf63e37354ba2cb6e7bec7fbe39fa3e70230015 project=default
t=2023-03-04T03:25:03+0000 lvl=info msg="No members have image, nothing to do" fingerprint=313f9ad6d74524b0d13f4ed9fbf63e37354ba2cb6e7bec7fbe39fa3e70230015 project=default
t=2023-03-04T03:25:03+0000 lvl=info msg="Syncing image to members finished" fingerprint=313f9ad6d74524b0d13f4ed9fbf63e37354ba2cb6e7bec7fbe39fa3e70230015 project=default
t=2023-03-04T03:26:48+0000 lvl=info msg="Creating container" ephemeral=false instance=chdb-container instanceType=container project=default
t=2023-03-04T03:26:50+0000 lvl=info msg="Created container" ephemeral=false instance=chdb-container instanceType=container project=default
t=2023-03-04T03:27:26+0000 lvl=info msg="Starting container" action=start created=2023-03-04T03:26:47+0000 ephemeral=false instance=chdb-container instanceType=container project=default stateful=false used=1970-01-01T00:00:00+0000
t=2023-03-04T03:27:27+0000 lvl=eror msg="The start hook failed" err="Failed to read template file: open /var/snap/lxd/common/lxd/containers/chdb-container/templates/hostname.tpl: no such file or directory" instance=chdb-container
t=2023-03-04T03:27:53+0000 lvl=eror msg="Failed starting container" action=start created=2023-03-04T03:26:47+0000 ephemeral=false instance=chdb-container instanceType=container project=default stateful=false used=1970-01-01T00:00:00+0000
t=2023-03-04T03:37:15+0000 lvl=info msg="Starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T03:17:26+0000
t=2023-03-04T03:37:15+0000 lvl=eror msg="Failed starting container" action=start created=2023-03-04T00:26:25+0000 ephemeral=false instance=ABC-Container instanceType=container project=default stateful=false used=2023-03-04T03:17:26+0000
t=2023-03-04T03:50:27+0000 lvl=info msg="Pruning expired instance backups"
t=2023-03-04T03:50:27+0000 lvl=info msg="Done pruning expired instance backups"
t=2023-03-04T04:50:27+0000 lvl=info msg="Pruning expired instance backups"
t=2023-03-04T04:50:27+0000 lvl=info msg="Done pruning expired instance backups"
I’ve read a couple of questions like this and they end up posting a lot of data. I didn’t want to start by posting yards of possibly irrelevant data. But obviously will on request.
Any help greatly appreciated.
Update:
I used lxc ‘launch image’ to create a new Debian 11 container and that worked just fine. I then ran the same log query to see if it would yield any useful clues. The new Debian 11 container log contained some of the same warnings as the imported container logs, which I thought were related to the accompanying errors; apparently not:
root# lxc info --show-log Debian11-Container
Name: Debian11-Container
Location: none
Remote: unix://
Architecture: x86_64
Created: 2023/03/04 23:43 UTC
Status: Running
Type: container
Profiles: default
Pid: 19746
Ips:
lo: inet 127.0.0.1
lo: inet6 ::1
Resources:
Processes: 6
Disk usage:
root: 13.43MB
CPU usage:
CPU usage (in seconds): 4
Memory usage:
Memory (current): 55.19MB
Memory (peak): 81.78MB
Network usage:
lo:
Bytes received: 0B
Bytes sent: 0B
Packets received: 0
Packets sent: 0
Log:
lxc Debian11-Container 20230304235757.214 WARN conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc Debian11-Container 20230304235757.214 WARN conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc Debian-Container 20230304235757.215 WARN conf - conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc Debian11 20230304235757.215 WARN conf - conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc Debian11 20230304235757.216 WARN cgfsng - cgroups/cgfsng.c:fchowmodat:1252 - No such file or directory - Failed to fchownat(40, memory.oom.group, 1000000000, 0, AT_EMPTY_PATH | AT_SYMLINK_NOFOLLOW )