Lxc commands fails with socket error on LXD 3.14

LXD
21

so the error messages have gone…

but the container doesnt start:

michael@indianer2:~$ lxc info --show-log webserver
Name: webserver
Location: none
Remote: unix://
Architecture: x86_64
Created: 2018/09/26 20:06 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc webserver 20190630005254.293 WARN     conf - conf.c:lxc_map_ids:2970 - newuidmap binary is missing
lxc webserver 20190630005254.293 WARN     conf - conf.c:lxc_map_ids:2976 - newgidmap binary is missing
lxc webserver 20190630005254.349 WARN     conf - conf.c:lxc_map_ids:2970 - newuidmap binary is missing
lxc webserver 20190630005254.349 WARN     conf - conf.c:lxc_map_ids:2976 - newgidmap binary is missing
lxc webserver 20190630005254.583 WARN     conf - conf.c:lxc_setup_devpts:1641 - Invalid argument - Failed to unmount old devpts instance
lxc webserver 20190630005254.925 WARN     conf - conf.c:lxc_map_ids:2970 - newuidmap binary is missing
lxc webserver 20190630005254.925 WARN     conf - conf.c:lxc_map_ids:2976 - newgidmap binary is missing
22 
michael@indianer2:~$ lxc list
+------------+---------+----------------------+-----------------------------------------------+------------+-----------+
|    NAME    |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
+------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| dbserver   | RUNNING | 192.168.1.230 (eth0) | 2001:16b8:30d4:ef00:216:3eff:fe11:c7b8 (eth0) | PERSISTENT | 8         |
+------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| mailserver | RUNNING | 192.168.1.220 (eth0) | 2001:16b8:30d4:ef00:216:3eff:fe59:303f (eth0) | PERSISTENT | 8         |
+------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| mailtools  | RUNNING | 192.168.1.240 (eth0) | 2001:16b8:30d4:ef00:216:3eff:fedb:e634 (eth0) | PERSISTENT |           |
+------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| webserver  | STOPPED |                      |                                               | PERSISTENT |           |
+------------+---------+----------------------+-----------------------------------------------+------------+-----------+
23

lxc console --show-log webserver

24 
Console log:

/sbin/init: error while loading shared libraries: libseccomp.so.2: cannot open shared object file: No such file or directory

is it possible that my conainer is damaged through the power loss?

25

can i somehow recover a snapshot?

26

Yeah, so I would do:

  • lxc snapshot webserver pre-restore
  • lxc restore webserver NAME-OF-MOST-RECENT-SNAPSHOT
27

Snapshot list is visible in lxc info webserver.

The console output above shows that there’s something wrong with that container’s init system, missing library or corrupted init somehow. May be fixable with some more work, but if you have a recent snapshot, restore would be easiest.

28 
michael@indianer2:~$ lxc snapshot webserver pre-restore
Error: subvolume snapshot failed, source=/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver, dest=/var/snap/lxd/common/lxd/storage-pools/default/containers-snapshots/webserver/pre-restore, output=ERROR: not a subvolume: /var/snap/lxd/common/lxd/storage-pools/default/containers/webserver

sorry for all these problems… :-/

29

Ok, I wonder if we can make a backup of it with:

  • lxc export webserver --container-only

That should get you a tarball of whatever is currently on disk, then you can do the lxc restore knowing you have a copy of the data that was there in case something goes bad.

30

this also didnt work:

michael@indianer2:~$ lxc export webserver --container-only
Error: Create backup: Backup storage: subvolume snapshot failed, source=/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver, dest=/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver856524048/.backup, output=ERROR: not a subvolume: /var/snap/lxd/common/lxd/storage-pools/default/containers/webserver
31

Okay, fine…

  • sudo mv
    /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver.bak
  • sudo mkdir /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pools/default/containers/webserver
  • lxc restore webserver SNAPSHOT-NAME
32 
michael@indianer2:~$ lxc restore webserver snap6
Error: not found

ok…should it be named michael@indianer2:~$ lxc restore webserver/snap6?

33

nope, lxc restore webserver snap6 is right… can you try with snap5?

Ownership of snap6 looked a bit odd in previous listings.

34

negative…:

michael@indianer2:~$ lxc restore webserver snap5
Error: not found
michael@indianer2:~$ lxc restore webserver snap4
Error: not found
michael@indianer2:~$ lxc restore webserver snap3
Error: not found
michael@indianer2:~$ lxc restore webserver snap2
Error: not found
35

Can you show:

  • lxc info webserver
  • ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/snapshots
  • ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/snapshots/webserver
  • ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/storage-pool/default/containers-snapshots
  • ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/storage-pool/default/containers-snapshots/webserver
36

lxc info webserver

Name: webserver
Location: none
Remote: unix://
Architecture: x86_64
Created: 2018/09/26 20:06 UTC
Status: Stopped
Type: persistent
Profiles: default

sudo ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/snapshots

ls: cannot access '/var/snap/lxd/common/mntns/var/snap/lxd/common/snapshots': No such file or directory

sudo ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/snapshots/webserver

ls: cannot access '/var/snap/lxd/common/mntns/var/snap/lxd/common/snapshots/webserver': No such file or directory

sudo ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/storage-pool/default/containers-snapshots

ls: cannot access '/var/snap/lxd/common/mntns/var/snap/lxd/common/storage-pool/default/containers-snapshots': No such file or directory

sudo ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/storage-pool/default/containers-snapshots/webserver

ls: cannot access '/var/snap/lxd/common/mntns/var/snap/lxd/common/storage-pool/default/containers-snapshots/webserver': No such file or directory
37

Hmm, lxc info suggests there are no snapshots in the database for that container.
That seems odd given the state of the filesystem.

lxd sql global "SELECT * FROM containers;"

38 
michael@indianer2:~$ lxd sql global "SELECT * FROM containers;"
+----+---------+---------------------+--------------+------+-----------+-------------------------------------+----------+-------------------------------------+-------------+------------+----------------------+
| id | node_id |        name         | architecture | type | ephemeral |            creation_date            | stateful |            last_use_date            | description | project_id |     expiry_date      |
+----+---------+---------------------+--------------+------+-----------+-------------------------------------+----------+-------------------------------------+-------------+------------+----------------------+
| 1  | 1       | webserver           | 2            | 0    | 0         | 2018-09-26T22:06:19+02:00           | 0        | 2019-06-30T02:52:53.911940505+02:00 |             | 1          | 0001-01-01T00:00:00Z |
| 2  | 1       | mailserver          | 2            | 0    | 0         | 2018-09-26T22:10:35+02:00           | 0        | 2019-06-30T02:40:03.209976254+02:00 |             | 1          | 0001-01-01T00:00:00Z |
| 3  | 1       | mailtools           | 2            | 0    | 0         | 2018-09-27T08:52:25+02:00           | 0        | 2019-06-30T02:40:04.483494926+02:00 |             | 1          | 0001-01-01T00:00:00Z |
| 4  | 1       | dbserver            | 2            | 0    | 0         | 2018-09-27T13:34:36+02:00           | 0        | 2019-06-30T02:40:01.949872431+02:00 |             | 1          | 0001-01-01T00:00:00Z |
| 29 | 1       | dbserver/20181108   | 2            | 1    | 0         | 2018-11-08T21:22:19.521426735+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 30 | 1       | mailserver/20181108 | 2            | 1    | 0         | 2018-11-08T21:36:57.46488988+01:00  | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 32 | 1       | dbserver/snap0      | 2            | 1    | 0         | 2018-11-20T10:47:07.368292616+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 33 | 1       | mailserver/snap0    | 2            | 1    | 0         | 2018-11-20T10:47:47.494293777+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 36 | 1       | dbserver/snap1      | 2            | 1    | 0         | 2018-11-27T11:58:50.509550761+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 37 | 1       | mailserver/snap1    | 2            | 1    | 0         | 2018-11-27T11:59:08.227095902+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 40 | 1       | dbserver/snap2      | 2            | 1    | 0         | 2018-12-19T15:47:34.830935276+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 41 | 1       | mailserver/snap2    | 2            | 1    | 0         | 2018-12-19T15:47:51.489952319+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 45 | 1       | dbserver/snap3      | 2            | 1    | 0         | 2019-02-05T11:11:00.587335385+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 46 | 1       | mailserver/snap3    | 2            | 1    | 0         | 2019-02-05T11:11:16.879600768+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 49 | 1       | dbserver/snap4      | 2            | 1    | 0         | 2019-02-19T17:26:24.91316927+01:00  | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 50 | 1       | dbserver/snap5      | 2            | 1    | 0         | 2019-02-26T10:56:52.229374658+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 51 | 1       | mailserver/snap4    | 2            | 1    | 0         | 2019-02-26T10:57:08.187580399+01:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 54 | 1       | mailserver/snap5    | 2            | 1    | 0         | 2019-04-17T10:01:30.970999331+02:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 55 | 1       | dbserver/snap6      | 2            | 1    | 0         | 2019-05-13T10:48:54.629995947+02:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
| 56 | 1       | mailserver/snap6    | 2            | 1    | 0         | 2019-05-13T10:49:11.351384668+02:00 | 0        | 1970-01-01T01:00:00+01:00           |             | 1          | 0001-01-01T00:00:00Z |
+----+---------+---------------------+--------------+------+-----------+-------------------------------------+----------+-------------------------------------+-------------+------------+----------------------+
39

seems like there is completely no snapshot for webserver…

40

Yeah, looks like those snapshots you have on disk for the webserver aren’t in the database.

  • ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/snapshots
  • ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/snapshots/webserver
  • ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pool/default/containers-snapshots
  • ls -lh /var/snap/lxd/common/mntns/var/snap/lxd/common/lxd/storage-pool/default/containers-snapshots/webserver