TF+LXD+snap, volume not mounted

LXD
,
1

I’m using LXC through TF and LXD, and also with the difference that LXD installed via snap means, and I’m new to all of them, so please be patient with me.

I have a container with 4 volumes:

$ sudo lxc list
+-------------+---------+------------------------+------+------------+-----------+----------+
|    NAME     |  STATE  |          IPV4          | IPV6 |    TYPE    | SNAPSHOTS | LOCATION |
+-------------+---------+------------------------+------+------------+-----------+----------+
| demo3-node1 | RUNNING | 10.254.254.231 (eth0)  |      | PERSISTENT | 0         | nimbus   |
|             |         | 10.230.230.1 (eth1)    |      |            |           |          |
|             |         | 10.0.3.154 (provision) |      |            |           |          |
+-------------+---------+------------------------+------+------------+-----------+----------+

$ sudo lxc storage volume list local
+-----------+-------------------------+-------------+---------+----------+
|   TYPE    |          NAME           | DESCRIPTION | USED BY | LOCATION |
+-----------+-------------------------+-------------+---------+----------+
| container | demo3-node1             |             | 1       | nimbus   |
+-----------+-------------------------+-------------+---------+----------+
| custom    | demo3-node1-cloudian001 |             | 1       | nimbus   |
+-----------+-------------------------+-------------+---------+----------+
| custom    | demo3-node1-cloudian002 |             | 1       | nimbus   |
+-----------+-------------------------+-------------+---------+----------+
| custom    | demo3-node1-cloudian003 |             | 1       | nimbus   |
+-----------+-------------------------+-------------+---------+----------+
| custom    | demo3-node1-cloudian004 |             | 1       | nimbus   |
+-----------+-------------------------+-------------+---------+----------+

The container’s config lists the volumes, but it seems they’re not attached?

$ sudo cat /var/snap/lxd/common/lxd/containers/demo3-node1/backup.yaml
container:
  architecture: x86_64
  config:
    boot.autostart: "1"
    image.architecture: x86_64
    image.description: Centos 7 x86_64 (20190325_07:08)
    image.name: centos-7-x86_64-default-20190325_07:08
    image.os: centos
    image.release: "7"
    image.serial: "20190325_07:08"
    image.variant: default
    limits.cpu: "4"
    limits.memory: 8GB
    volatile.base_image: 2d8190b364998ba6edfbcd08509ffce3433f8e84c864a394e9f5c305bacf52f8
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
  devices:
    cloudian001:
      path: /cloudian001
      pool: local
      source: demo3-node1-cloudian001
      type: disk
    cloudian002:
      path: /cloudian002
      pool: local
      source: demo3-node1-cloudian002
      type: disk
    cloudian003:
      path: /cloudian003
      pool: local
      source: demo3-node1-cloudian003
      type: disk
    cloudian004:
      path: /cloudian004
      pool: local
      source: demo3-node1-cloudian004
      type: disk
  ephemeral: false
  profiles:
  - default
  stateful: false
  description: ""
  created_at: 2019-12-18T12:28:18-03:00
  expanded_config:
    boot.autostart: "1"
    image.architecture: x86_64
    image.description: Centos 7 x86_64 (20190325_07:08)
    image.name: centos-7-x86_64-default-20190325_07:08
    image.os: centos
    image.release: "7"
    image.serial: "20190325_07:08"
    image.variant: default
    limits.cpu: "4"
    limits.memory: 8GB
    volatile.base_image: 2d8190b364998ba6edfbcd08509ffce3433f8e84c864a394e9f5c305bacf52f8
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
  expanded_devices:
    cloudian001:
      path: /cloudian001
      pool: local
      source: demo3-node1-cloudian001
      type: disk
    cloudian002:
      path: /cloudian002
      pool: local
      source: demo3-node1-cloudian002
      type: disk
    cloudian003:
      path: /cloudian003
      pool: local
      source: demo3-node1-cloudian003
      type: disk
    cloudian004:
      path: /cloudian004
      pool: local
      source: demo3-node1-cloudian004
  type: disk
    root:
      path: /
      pool: local
      type: disk
  name: demo3-node1
  status: Running
  status_code: 103
  last_used_at: 2019-12-18T12:28:23.241561778-03:00
  location: nimbus
snapshots: []
pool:
  config:
    source: /var/snap/lxd/common/lxd/storage-pools/local
  description: ""
  name: local
  driver: dir
  used_by: []
  status: Created
  locations:
  - nimbus
volume:
  config: {}
  description: ""
  name: demo3-node1
  type: container
  used_by: []
  location: nimbus

The container’s config file also mentions them:

$ sudo cat /var/snap/lxd/common/lxd/logs/demo3-node1/lxc.conf
lxc.mount.auto = proc:rw sys:rw
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/net/tun dev/net/tun none bind,create=file,optional 0 0
lxc.mount.entry = /proc/sys/fs/binfmt_misc proc/sys/fs/binfmt_misc none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/firmware/efi/efivars sys/firmware/efi/efivars none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/fuse/connections sys/fs/fuse/connections none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/fs/pstore sys/fs/pstore none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = /sys/kernel/security sys/kernel/security none rbind,create=dir,optional 0 0
lxc.mount.entry = /dev/mqueue dev/mqueue none rbind,create=dir,optional 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devlxd dev/lxd none bind,create=dir 0 0
lxc.rootfs.path = dir:/var/snap/lxd/common/lxd/containers/demo3-node1/rootfs
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/demo3-node1/disk.cloudian001.cloudian001 cloudian001 none bind,create=dir 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/demo3-node1/disk.cloudian002.cloudian002 cloudian002 none bind,create=dir 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/demo3-node1/disk.cloudian003.cloudian003 cloudian003 none bind,create=dir 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/devices/demo3-node1/disk.cloudian004.cloudian004 cloudian004 none bind,create=dir 0 0
lxc.mount.entry = /var/snap/lxd/common/lxd/shmounts/demo3-node1 dev/.lxd-mounts none bind,create=dir 0 0

I found these two directories named after the volume:

/var/snap/lxd/common/lxd/devices/demo3-node1/disk.cloudian001.cloudian001
/var/snap/lxd/common/lxd/storage-pools/local/custom/demo3-node1-cloudian001

but none is mounted on the host.

In the container I have:

[root@demo3-node1 ~]# cat /etc/fstab 
UUID=   /cloudian1      ext4    defaults,rw,nosuid,noexec,nodev,noatime,noinit_itable,data=ordered,errors=remount-ro    0       1
UUID=   /cloudian2      ext4    defaults,rw,nosuid,noexec,nodev,noatime,noinit_itable,data=ordered,errors=remount-ro    0       1
UUID=   /cloudian3      ext4    defaults,rw,nosuid,noexec,nodev,noatime,noinit_itable,data=ordered,errors=remount-ro    0       1
UUID=   /cloudian4      ext4    defaults,rw,nosuid,noexec,nodev,noatime,noinit_itable,data=ordered,errors=remount-ro    0       1

and:

[root@demo3-node1 ~]# mount | grep cloudian | wc -l
0

That means none is mounted in the container either.

So I thought “Maybe they’re not attached. That’s easy!”, but:

$ sudo lxc storage volume attach local demo3-node1-cloudian001 demo3-node1 /cloudian001
Error: Invalid devices: More than one disk device uses the same path: /cloudian001

At this point, I don’t know what else to do. I mentioned that I’m using Terraform; it’s with this backend:

I can provide the main.tf, but I’m not really sure it’s relevant.

2

I just found out that the volumes are mounted at the beginning:

/dev/nvme0n1p3 on /cloudian001 type ext4 (rw,relatime,errors=remount-ro)
/dev/nvme0n1p3 on /cloudian002 type ext4 (rw,relatime,errors=remount-ro)
/dev/nvme0n1p3 on /cloudian003 type ext4 (rw,relatime,errors=remount-ro)
/dev/nvme0n1p3 on /cloudian004 type ext4 (rw,relatime,errors=remount-ro)

But for some reason they disappear later. I’m reading https://github.com/lxc/lxc/issues/2798 to see if it applies to me, the problem looks like similar.

3

I fixed this, our provisioning system was not prepared for running in containers.