LVM storage problem - all container stopped and doesn't start anymore after switch to core20

esosan · June 17, 2021, 6:50am

Hello, I have a huge problem: all containers stopped and doesn’t start anymore.

When I try to start a container I have this error:

Error: Failed preparing container for start: Failed to activate LVM logical volume "/dev/pool_0/containers_iceCast": Failed to run: lvchange --activate y --ignoreactivationskip /dev/pool_0/containers_iceCast: Volume group "pool_0" not found
  Cannot process volume group pool_0

But with vgs/lvs commands I see vg and lv volumes.

Some one have experienced this kind of troubles?

esosan

(I have also filled a issue on GitHub)

tomp · June 17, 2021, 8:16am

Please can you show the output of:

sudo lvs

lxc storage ls
lxc storage show <lvm pool name>

Also, what version of LXD are you running?

esosan · June 17, 2021, 8:26am

hi tomp, I’m using 4.15.
here the outputs

  containers_gdg            pool_0    -wi------k   3.00g                                                    
  containers_git            pool_0    -wi------k   3.00g                                                    
  containers_iceCast        pool_0    -wi------k   3.00g                                                    
  containers_m--clamav      pool_0    -wi------k   3.00g                                                    
  containers_m--clamavOLD   pool_0    -wi------k   3.00g                                                    
  containers_m--dcc         pool_0    -wi------k   3.00g                                                    
  containers_m--dcot        pool_0    -wi------k   3.00g                                                    
  containers_m--pfix        pool_0    -wi------k   3.00g                                                    
  containers_m--spam18      pool_0    -wi------k   3.00g                                                    
  containers_mysql          pool_0    -wi------k   3.00g                                                    
  containers_named          pool_0    -wi------k   3.00g                                                    
  containers_owncloud       pool_0    -wi------k   3.00g                                                    
  containers_rCube          pool_0    -wi------k   3.00g                                                    
  containers_rCube14        pool_0    -wi------k   3.00g                                                    
  containers_redis          pool_0    -wi------k   3.00g                                                    
  containers_solr           pool_0    -wi------k   3.00g                                                    
  containers_www--ext       pool_0    -wi------k   3.00g                                                    
  containers_www--std       pool_0    -wi------k   3.00g                                                    
  containers_wwwHurdega     pool_0    -wi------k   3.00g                                                    
  custom_default_git_data   pool_0    -wi------k   3.00g                                                    
  custom_default_data       pool_OC   -wi------k 193.00g                                                    
  custom_default_solrData   pool_data -wi------k <11.18g                                                    
  custom_default_vmail_data pool_data -wi------k <29.32g                                                    
  custom_default_www_audio  pool_data -wi------k   5.00g                                                    
  home                      vg        -wi-ao---- <19.53g                                                    
  lxc_pool00                vg        -wi-a-----  60.00g                                                    
  lxc_pool01                vg        -wi-a-----  20.00g                                                    
  lxc_poolOC00              vg        -wi-a----- 220.00g                                                    
  lxc_pooldata00            vg        -wi-a-----  50.00g                                                    
  lxc_pooldata01            vg        -wi-a-----  50.00g

lxc storage ls

+-----------+--------+------------------------------------------------+-------------+---------+
|   NAME    | DRIVER |                     SOURCE                     | DESCRIPTION | USED BY |
+-----------+--------+------------------------------------------------+-------------+---------+
| default   | dir    | /var/snap/lxd/common/lxd/storage-pools/default |             | 0       |
+-----------+--------+------------------------------------------------+-------------+---------+
| pool0     | lvm    | pool_0                                         |             | 21      |
+-----------+--------+------------------------------------------------+-------------+---------+
| poolOC    | lvm    | pool_OC                                        |             | 1       |
+-----------+--------+------------------------------------------------+-------------+---------+
| pool_DATA | lvm    | pool_data                                      |             | 3       |
+-----------+--------+------------------------------------------------+-------------+---------+

lxc storage show pool0

config:
  lvm.use_thinpool: "false"
  lvm.vg_name: pool_0
  source: pool_0
  volatile.initial_source: pool_0
  volume.block.filesystem: xfs
  volume.size: "3221225472"
description: ""
name: pool0
driver: lvm
used_by:
- /1.0/instances/gdg
- /1.0/instances/git
- /1.0/instances/iceCast
- /1.0/instances/m-clamav
- /1.0/instances/m-clamavOLD
- /1.0/instances/m-dcc
- /1.0/instances/m-dcot
- /1.0/instances/m-pfix
- /1.0/instances/m-spam18
- /1.0/instances/mysql
- /1.0/instances/named
- /1.0/instances/owncloud
- /1.0/instances/rCube
- /1.0/instances/rCube14
- /1.0/instances/redis
- /1.0/instances/solr
- /1.0/instances/www-ext
- /1.0/instances/www-std
- /1.0/instances/wwwHurdega
- /1.0/profiles/default
- /1.0/storage-pools/pool0/volumes/custom/git_data
status: Created
locations:
- none

tomp · June 17, 2021, 8:46am

And the output of sudo vgs too please.

tomp · June 17, 2021, 8:47am

Also can you try this:

sudo snap set lxd lvm.external=true
sudo systemctl reload snap.lxd.daemon

This will get LXD to use your host’s lvm tools rather than the bundled snap ones.

tomp · June 17, 2021, 8:48am

What host OS and version are you using btw?

esosan · June 17, 2021, 8:49am

  VG        #PV #LV #SN Attr   VSize    VFree  
  pool_0      2  20   0 wz--n-   79.99g  19.99g
  pool_OC     1   1   0 wz--n- <220.00g <27.00g
  pool_data   2   3   0 wz--n-   99.99g <54.50g
  vg          1   6   0 wz--n-   <1.80t  <1.39t

esosan · June 17, 2021, 8:52am

ubuntu 18.04

esosan · June 17, 2021, 8:53am

I have try, and now the containers are all running. but there is a new problem, no one have an assigned ip.

tomp · June 17, 2021, 8:55am

OK cool, well that’s interesting that that allowed them to start.
Can you show the output of sudo lvs again now, and also can you lxc shell <instance> ok into one of the containers?

esosan · June 17, 2021, 8:57am

  LV                        VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  containers_gdg            pool_0    -wi-ao---k   3.00g                                                    
  containers_git            pool_0    -wi-ao---k   3.00g                                                    
  containers_iceCast        pool_0    -wi-ao---k   3.00g                                                    
  containers_m--clamav      pool_0    -wi-ao---k   3.00g                                                    
  containers_m--clamavOLD   pool_0    -wi------k   3.00g                                                    
  containers_m--dcc         pool_0    -wi------k   3.00g                                                    
  containers_m--dcot        pool_0    -wi-ao---k   3.00g                                                    
  containers_m--pfix        pool_0    -wi-ao---k   3.00g                                                    
  containers_m--spam18      pool_0    -wi-ao---k   3.00g                                                    
  containers_mysql          pool_0    -wi-ao---k   3.00g                                                    
  containers_named          pool_0    -wi-ao---k   3.00g                                                    
  containers_owncloud       pool_0    -wi-ao---k   3.00g                                                    
  containers_rCube          pool_0    -wi-ao---k   3.00g                                                    
  containers_rCube14        pool_0    -wi------k   3.00g                                                    
  containers_redis          pool_0    -wi-ao---k   3.00g                                                    
  containers_solr           pool_0    -wi-ao---k   3.00g                                                    
  containers_www--ext       pool_0    -wi------k   3.00g                                                    
  containers_www--std       pool_0    -wi-ao---k   3.00g                                                    
  containers_wwwHurdega     pool_0    -wi-ao---k   3.00g                                                    
  custom_default_git_data   pool_0    -wi-ao---k   3.00g                                                    
  custom_default_data       pool_OC   -wi-ao---k 193.00g                                                    
  custom_default_solrData   pool_data -wi-ao---k <11.18g                                                    
  custom_default_vmail_data pool_data -wi-ao---k <29.32g                                                    
  custom_default_www_audio  pool_data -wi-ao---k   5.00g                                                    
  home                      vg        -wi-ao---- <19.53g                                                    
  lxc_pool00                vg        -wi-ao----  60.00g                                                    
  lxc_pool01                vg        -wi-ao----  20.00g                                                    
  lxc_poolOC00              vg        -wi-ao---- 220.00g                                                    
  lxc_pooldata00            vg        -wi-ao----  50.00g                                                    
  lxc_pooldata01            vg        -wi-a-----  50.00g

esosan · June 17, 2021, 8:58am

lxc shell works!

now it’s just the network the problem…

tomp · June 17, 2021, 9:00am

Can you show output of lxc config show <instance> --expanded for one of the problem instances.

Then also show lxc network show <network> for the network that the instance is connected to.

Also can you show the output of ip a and ip r from the LXD host and inside one of the problem instances.

esosan · June 17, 2021, 9:05am

ips are redacted…

architecture: x86_64
config:
  boot.autostart: "true"
  boot.autostart.priority: "1000"
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20190212.1)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20190212.1"
  image.version: "18.04"
  volatile.base_image: 35f6bff57c250208c6dc73445eefe8c1020853a1bc8571ebd8bfb1fe3a239504
  volatile.eth0.host_name: veth3515a489
  volatile.eth0.hwaddr: 00:16:3e:fa:70:ab
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: f5e3b6f5-7059-4995-999e-4c5841dfceee
devices:
  bind_4_tcp:
    bind: host
    connect: tcp:10.4.43.176:53
    listen: tcp:##.##.##.##:53
    nat: "true"
    type: proxy
  bind_4_udp:
    bind: host
    connect: udp:10.4.43.176:53
    listen: udp:##.##.##.##:53
    nat: "true"
    type: proxy
  bind_6_tcp:
    bind: host
    connect: tcp:[fd42:26ae:9c2f:e23:216:3eff:fefa:70ab]:53
    listen: tcp:[##:##:#:#::#]:53
    nat: "true"
    type: proxy
  bind_6_udp:
    bind: host
    connect: udp:[fd42:26ae:9c2f:e23:216:3eff:fefa:70ab]:53
    listen: udp:[##:##:#:#::#]:53
    nat: "true"
    type: proxy
  eth0:
    ipv4.address: 10.4.43.176
    ipv6.address: fd42:26ae:9c2f:e23:216:3eff:fefa:70ab
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: pool0
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

lxc network show lxdbr0
config:
ipv4.address: 10.4.43.1/24
ipv4.nat: “true”
ipv6.address: fd42:26ae:9c2f:e23::1/64
ipv6.nat: “true”
raw.dnsmasq: |
auth-zone=lxd
dns-loop-detect
description: “”
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/gdg
- /1.0/instances/git
- /1.0/instances/iceCast
- /1.0/instances/m-clamav
- /1.0/instances/m-clamavOLD
- /1.0/instances/m-dcc
- /1.0/instances/m-dcot
- /1.0/instances/m-pfix
- /1.0/instances/m-spam18
- /1.0/instances/mysql
- /1.0/instances/named
- /1.0/instances/owncloud
- /1.0/instances/rCube
- /1.0/instances/rCube14
- /1.0/instances/redis
- /1.0/instances/solr
- /1.0/instances/www-ext
- /1.0/instances/www-std
- /1.0/instances/wwwHurdega
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

tomp · June 17, 2021, 9:08am

Cool thanks. And what does sudo ps aux | grep dnsmasq show?

Also sudo ss -ulpn?

tomp · June 17, 2021, 9:15am

Can you also show sudo grep /var/snap/lxd/common/lxd/logs/lxd.log dnsmasq please

esosan · June 17, 2021, 9:15am

LXC HOST

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
link/ether 00:25:90:79:85:4a brd ff:ff:ff:ff:ff:ff
inet 46.105.121.211/24 brd 46.105.121.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet6 2001:41d0:2:e8d3::1/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::225:90ff:fe79:854a/64 scope link
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:25:90:79:85:4b brd ff:ff:ff:ff:ff:ff
4: wgclient_home: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.127.10.12/24 scope global wgclient_home
valid_lft forever preferred_lft forever
5: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:97:44:03 brd ff:ff:ff:ff:ff:ff
inet 10.4.43.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:26ae:9c2f:e23::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe97:4403/64 scope link
valid_lft forever preferred_lft forever
7: vethae7df84d@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 5a:72:a1:27:14:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
9: veth3515a489@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether ea:4d:8a:2c:91:14 brd ff:ff:ff:ff:ff:ff link-netnsid 1
11: veth11bb0b31@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 02:00:0e:6e:61:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 2
13: veth58fde0b5@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether e6:30:b7:ae:2f:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 3
15: veth982cdf44@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 36:82:ef:bb:05:c1 brd ff:ff:ff:ff:ff:ff link-netnsid 4
17: veth712d0c5b@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 4e:6b:12:f1:96:56 brd ff:ff:ff:ff:ff:ff link-netnsid 5
19: veth36f55459@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether fa:d4:b5:f3:55:52 brd ff:ff:ff:ff:ff:ff link-netnsid 6
21: veth4bd7e789@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether c6:bb:34:17:72:56 brd ff:ff:ff:ff:ff:ff link-netnsid 7
23: veth97903699@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 9a:1a:2f:66:9d:de brd ff:ff:ff:ff:ff:ff link-netnsid 8
25: vethcb7e14a3@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether a2:a2:a7:da:18:13 brd ff:ff:ff:ff:ff:ff link-netnsid 9
27: vethb07621eb@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 46:aa:78:14:c8:58 brd ff:ff:ff:ff:ff:ff link-netnsid 10
29: vethd6fe5eba@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 16:59:31:6a:1b:ca brd ff:ff:ff:ff:ff:ff link-netnsid 11
31: vetheb147d01@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 32:d5:14:1c:65:71 brd ff:ff:ff:ff:ff:ff link-netnsid 12
33: veth4c1f23ac@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether d6:1c:fa:c9:c1:2d brd ff:ff:ff:ff:ff:ff link-netnsid 13
35: vethfe8aae29@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 9e:16:f1:80:1a:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 14

ip r
default via ... dev enp1s0 proto static
10.4.43.0/24 dev lxdbr0 proto kernel scope link src 10.4.43.1
10.127.10.0/24 dev wgclient_home proto kernel scope link src 10.127.10.12
.../24 dev enp1s0 proto kernel scope link src 46.105.121.211
.../24 dev wgclient_home scope link

esosan · June 17, 2021, 9:17am

LXC INSTANCE

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:fa:70:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::216:3eff:fefa:70ab/64 scope link
valid_lft forever preferred_lft forever

ip r
nothing

esosan · June 17, 2021, 9:19am

sudo ps aux | grep dnsmasq
root 7438 0.0 0.0 13144 1096 pts/0 S+ 11:17 0:00 grep --color=auto dnsmasq

it’s not started?

ss -ulpn
    State     Recv-Q      Send-Q            Local Address:Port            Peer Address:Port                                                                                                                                                                                                        
    UNCONN    0           0                       0.0.0.0:58406                0.0.0.0:*                                                                                                                                                                                                           
    UNCONN    0           0                     127.0.0.1:53                   0.0.0.0:*         users:(("named",pid=1921,fd=518),("named",pid=1921,fd=517),("named",pid=1921,fd=516),("named",pid=1921,fd=515),("named",pid=1921,fd=514),("named",pid=1921,fd=513),("named",pid=1921,fd=512))     
    UNCONN    0           0                 127.0.0.53%lo:53                   0.0.0.0:*         users:(("systemd-resolve",pid=1647,fd=12))                                                                                                                                                        
    UNCONN    0           0                          [::]:58406                   [::]:*                                                                                                                                                                                                           
    UNCONN    0           0                         [::1]:53                      [::]:*         users:(("named",pid=1921,fd=525),("named",pid=1921,fd=524),("named",pid=1921,fd=523),("named",pid=1921,fd=522),("named",pid=1921,fd=521),("named",pid=1921,fd=520),("named",pid=1921,fd=519))

esosan · June 17, 2021, 9:20am

grep dnsmasq /var/snap/lxd/common/lxd/logs/lxd.log
t=2021-06-17T10:50:21+0200 lvl=warn msg=“Skipping AppArmor for dnsmasq due to raw.dnsmasq being set” driver=bridge name=lxdbr0 network=lxdbr0 project=default
t=2021-06-17T10:50:22+0200 lvl=eror msg=“The dnsmasq process exited prematurely” driver=bridge err=“Process exited with non-zero value 1” network=lxdbr0 project=default