Can't mount external XFS device/mountpoint in container

LXD
,
#1

Hi.

After snap updated to version 5.3 LXD started to get weird, even lxc ls got super slow.

installed:          5.3-924be6a              (23243) 107MB -

But I would like a solution to the XFS problem, after a reboot on the host I couldn’t mount this XFS device inside the container, I managed to change it to another FS and it mounted without problems.

I got this error:

Error: Failed to run: /snap/lxd/current/bin/lxd forkstart mongo-tests /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/mongo-tests/lxc.conf:
Try lxc info --show-log mongo-tests for more info

Log:

lxc mongo-tests 20220701115940.639 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc mongo-tests 20220701115940.639 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc mongo-tests 20220701115940.640 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc mongo-tests 20220701115940.640 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc mongo-tests 20220701115940.779 ERROR    conf - ../src/src/lxc/conf.c:mount_entry:2459 - Operation not permitted - Failed to mount "/var/snap/lxd/common/lxd/devices/mongo-tests/disk.MONGO.var-lib-mongodb" on "/var/snap/lxd/common/lxc//var/lib/mongodb"
lxc mongo-tests 20220701115940.779 ERROR    conf - ../src/src/lxc/conf.c:lxc_setup:4375 - Failed to setup mount entries
lxc mongo-tests 20220701115940.779 ERROR    start - ../src/src/lxc/start.c:do_start:1275 - Failed to setup container "mongo-tests"
lxc mongo-tests 20220701115940.779 ERROR    sync - ../src/src/lxc/sync.c:sync_wait:34 - An error occurred in another process (expected sequence number 3)
lxc mongo-tests 20220701115940.783 WARN     network - ../src/src/lxc/network.c:lxc_delete_network_priv:3631 - Failed to rename interface with index 0 from "eth0" to its initial name "vethbb155270"
lxc mongo-tests 20220701115940.783 ERROR    start - ../src/src/lxc/start.c:__lxc_start:2074 - Failed to spawn container "mongo-tests"
lxc mongo-tests 20220701115940.783 ERROR    lxccontainer - ../src/src/lxc/lxccontainer.c:wait_on_daemonized_start:877 - Received container state "ABORTING" instead of "RUNNING"
lxc mongo-tests 20220701115940.783 WARN     start - ../src/src/lxc/start.c:lxc_abort:1039 - No such process - Failed to send SIGKILL via pidfd 17 for process 795199
lxc mongo-tests 20220701115940.930 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3592 - newuidmap binary is missing
lxc mongo-tests 20220701115940.930 WARN     conf - ../src/src/lxc/conf.c:lxc_map_ids:3598 - newgidmap binary is missing
lxc 20220701115940.970 ERROR    af_unix - ../src/src/lxc/af_unix.c:lxc_abstract_unix_recv_fds_iov:218 - Connection reset by peer - Failed to receive response
lxc 20220701115940.970 ERROR    commands - ../src/src/lxc/commands.c:lxc_cmd_rsp_recv_fds:127 - Failed to receive file descriptors for command "get_state"

Any ideas?

#2

Looks like the same issue as

and

#3

Please could you enable the debug log using:

sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon

And then when running lxc ls when it gets slow and get us a copy of the file /var/snap/lxd/common/lxd/logs/lxd.log that would be great, thank you.

#4

Yesterday I went to try a systemctl reload snap.lxd.daemon and the LXD just stopped working starting Error: LXD is shutting down and with that forced a Reboot to host.

This the best I can do in this server:

lxc monitor --pretty

WARNING[2022-07-01T13:12:48Z] Transaction timed out. Retrying once          err="Failed to begin transaction: context deadline exceeded" member=1
DEBUG  [2022-07-01T13:13:11Z] Database error                                err="sql: transaction has already been committed or rolled back"
WARNING[2022-07-01T13:13:11Z] Failed to rollback transaction after error (Failed to fetch from "config" table: sql: Rows are closed): sql: transaction has already been committed or rolled back 
DEBUG  [2022-07-01T13:13:11Z] Database error                                err="Failed to fetch from \"config\" table: sql: Rows are closed"
DEBUG  [2022-07-01T13:13:11Z] Dqlite: attempt 1: server 1: connected 
WARNING[2022-07-01T13:13:21Z] Transaction timed out. Retrying once          err="Failed to begin transaction: context deadline exceeded" member=1
WARNING[2022-07-01T13:13:39Z] Transaction timed out. Retrying once          err="Failed to begin transaction: context deadline exceeded" member=1

time lxc ls -c ntspNmD4uSac
real	1m40.306s
user	0m0.326s
sys	0m0.170s
#5

Can you run

sudo snap set lxd daemon.debug=true; sudo systemctl reload snap.lxd.daemon

or reboot the host after running sudo snap set lxd daemon.debug=true;?

#6

Sorry, I can’t reboot this server now or have a risk to failure.

#7

Thanks, when you can will be great, as so far no-one has been able to provide debug logs, and we’ve not been able to reproduce locally, meaning we are not progressing toward a fix currently.

#8

My problem here is not the fact lxc ls is slow.
I was able to create in other server same issue with XFS filesystem, but in this server LXD run just a few containers (so I can reboot host if necessary) and lxc ls still normal output speed, running on LXD 5.3.

I did

sudo snap set lxd daemon.debug=true; 
sudo systemctl reload snap.lxd.daemon

The entire log have 2.5m

#9

Thanks, can you send it to me please using thomas.parrott at canonical.com

What is the specific problem with that host then?

#10

E-mail sent.

#11

Problem is:
LXD is not mounting XFS device inside container.

lxc config edit mongo-tests

  MONGO:
    path: /var/lib/mongodb
    source: /mnt/mongodata/mongo-tests/
    type: disk
#12

Sounds like a duplicate of

#13

Latest version (5.3-91e042b) fixed this problem (mounting) in my test-server.
Some important servers still haven’t refreshed snap version, and lxd ls, lxc info is slow and showing bunch of errors, but containers are running fine.