LVM storage problem - all container stopped and doesn't start anymore after switch to core20

OK cool, well that’s interesting that that allowed them to start.
Can you show the output of sudo lvs again now, and also can you lxc shell <instance> ok into one of the containers?

  LV                        VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  containers_gdg            pool_0    -wi-ao---k   3.00g                                                    
  containers_git            pool_0    -wi-ao---k   3.00g                                                    
  containers_iceCast        pool_0    -wi-ao---k   3.00g                                                    
  containers_m--clamav      pool_0    -wi-ao---k   3.00g                                                    
  containers_m--clamavOLD   pool_0    -wi------k   3.00g                                                    
  containers_m--dcc         pool_0    -wi------k   3.00g                                                    
  containers_m--dcot        pool_0    -wi-ao---k   3.00g                                                    
  containers_m--pfix        pool_0    -wi-ao---k   3.00g                                                    
  containers_m--spam18      pool_0    -wi-ao---k   3.00g                                                    
  containers_mysql          pool_0    -wi-ao---k   3.00g                                                    
  containers_named          pool_0    -wi-ao---k   3.00g                                                    
  containers_owncloud       pool_0    -wi-ao---k   3.00g                                                    
  containers_rCube          pool_0    -wi-ao---k   3.00g                                                    
  containers_rCube14        pool_0    -wi------k   3.00g                                                    
  containers_redis          pool_0    -wi-ao---k   3.00g                                                    
  containers_solr           pool_0    -wi-ao---k   3.00g                                                    
  containers_www--ext       pool_0    -wi------k   3.00g                                                    
  containers_www--std       pool_0    -wi-ao---k   3.00g                                                    
  containers_wwwHurdega     pool_0    -wi-ao---k   3.00g                                                    
  custom_default_git_data   pool_0    -wi-ao---k   3.00g                                                    
  custom_default_data       pool_OC   -wi-ao---k 193.00g                                                    
  custom_default_solrData   pool_data -wi-ao---k <11.18g                                                    
  custom_default_vmail_data pool_data -wi-ao---k <29.32g                                                    
  custom_default_www_audio  pool_data -wi-ao---k   5.00g                                                    
  home                      vg        -wi-ao---- <19.53g                                                    
  lxc_pool00                vg        -wi-ao----  60.00g                                                    
  lxc_pool01                vg        -wi-ao----  20.00g                                                    
  lxc_poolOC00              vg        -wi-ao---- 220.00g                                                    
  lxc_pooldata00            vg        -wi-ao----  50.00g                                                    
  lxc_pooldata01            vg        -wi-a-----  50.00g

lxc shell works!

now it’s just the network the problem…

Can you show output of lxc config show <instance> --expanded for one of the problem instances.

Then also show lxc network show <network> for the network that the instance is connected to.

Also can you show the output of ip a and ip r from the LXD host and inside one of the problem instances.

ips are redacted…

architecture: x86_64
config:
  boot.autostart: "true"
  boot.autostart.priority: "1000"
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20190212.1)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20190212.1"
  image.version: "18.04"
  volatile.base_image: 35f6bff57c250208c6dc73445eefe8c1020853a1bc8571ebd8bfb1fe3a239504
  volatile.eth0.host_name: veth3515a489
  volatile.eth0.hwaddr: 00:16:3e:fa:70:ab
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
  volatile.uuid: f5e3b6f5-7059-4995-999e-4c5841dfceee
devices:
  bind_4_tcp:
    bind: host
    connect: tcp:10.4.43.176:53
    listen: tcp:##.##.##.##:53
    nat: "true"
    type: proxy
  bind_4_udp:
    bind: host
    connect: udp:10.4.43.176:53
    listen: udp:##.##.##.##:53
    nat: "true"
    type: proxy
  bind_6_tcp:
    bind: host
    connect: tcp:[fd42:26ae:9c2f:e23:216:3eff:fefa:70ab]:53
    listen: tcp:[##:##:#:#::#]:53
    nat: "true"
    type: proxy
  bind_6_udp:
    bind: host
    connect: udp:[fd42:26ae:9c2f:e23:216:3eff:fefa:70ab]:53
    listen: udp:[##:##:#:#::#]:53
    nat: "true"
    type: proxy
  eth0:
    ipv4.address: 10.4.43.176
    ipv6.address: fd42:26ae:9c2f:e23:216:3eff:fefa:70ab
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: pool0
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

lxc network show lxdbr0
config:
ipv4.address: 10.4.43.1/24
ipv4.nat: “true”
ipv6.address: fd42:26ae:9c2f:e23::1/64
ipv6.nat: “true”
raw.dnsmasq: |
auth-zone=lxd
dns-loop-detect
description: “”
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/gdg
- /1.0/instances/git
- /1.0/instances/iceCast
- /1.0/instances/m-clamav
- /1.0/instances/m-clamavOLD
- /1.0/instances/m-dcc
- /1.0/instances/m-dcot
- /1.0/instances/m-pfix
- /1.0/instances/m-spam18
- /1.0/instances/mysql
- /1.0/instances/named
- /1.0/instances/owncloud
- /1.0/instances/rCube
- /1.0/instances/rCube14
- /1.0/instances/redis
- /1.0/instances/solr
- /1.0/instances/www-ext
- /1.0/instances/www-std
- /1.0/instances/wwwHurdega
- /1.0/profiles/default
managed: true
status: Created
locations:
- none

Cool thanks. And what does sudo ps aux | grep dnsmasq show?

Also sudo ss -ulpn?

Can you also show sudo grep /var/snap/lxd/common/lxd/logs/lxd.log dnsmasq please

LXC HOST

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
link/ether 00:25:90:79:85:4a brd ff:ff:ff:ff:ff:ff
inet 46.105.121.211/24 brd 46.105.121.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet6 2001:41d0:2:e8d3::1/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::225:90ff:fe79:854a/64 scope link
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:25:90:79:85:4b brd ff:ff:ff:ff:ff:ff
4: wgclient_home: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.127.10.12/24 scope global wgclient_home
valid_lft forever preferred_lft forever
5: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:97:44:03 brd ff:ff:ff:ff:ff:ff
inet 10.4.43.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:26ae:9c2f:e23::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe97:4403/64 scope link
valid_lft forever preferred_lft forever
7: vethae7df84d@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 5a:72:a1:27:14:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
9: veth3515a489@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether ea:4d:8a:2c:91:14 brd ff:ff:ff:ff:ff:ff link-netnsid 1
11: veth11bb0b31@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 02:00:0e:6e:61:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 2
13: veth58fde0b5@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether e6:30:b7:ae:2f:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 3
15: veth982cdf44@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 36:82:ef:bb:05:c1 brd ff:ff:ff:ff:ff:ff link-netnsid 4
17: veth712d0c5b@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 4e:6b:12:f1:96:56 brd ff:ff:ff:ff:ff:ff link-netnsid 5
19: veth36f55459@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether fa:d4:b5:f3:55:52 brd ff:ff:ff:ff:ff:ff link-netnsid 6
21: veth4bd7e789@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether c6:bb:34:17:72:56 brd ff:ff:ff:ff:ff:ff link-netnsid 7
23: veth97903699@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 9a:1a:2f:66:9d:de brd ff:ff:ff:ff:ff:ff link-netnsid 8
25: vethcb7e14a3@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether a2:a2:a7:da:18:13 brd ff:ff:ff:ff:ff:ff link-netnsid 9
27: vethb07621eb@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 46:aa:78:14:c8:58 brd ff:ff:ff:ff:ff:ff link-netnsid 10
29: vethd6fe5eba@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 16:59:31:6a:1b:ca brd ff:ff:ff:ff:ff:ff link-netnsid 11
31: vetheb147d01@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 32:d5:14:1c:65:71 brd ff:ff:ff:ff:ff:ff link-netnsid 12
33: veth4c1f23ac@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether d6:1c:fa:c9:c1:2d brd ff:ff:ff:ff:ff:ff link-netnsid 13
35: vethfe8aae29@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 9e:16:f1:80:1a:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 14

ip r
default via ... dev enp1s0 proto static
10.4.43.0/24 dev lxdbr0 proto kernel scope link src 10.4.43.1
10.127.10.0/24 dev wgclient_home proto kernel scope link src 10.127.10.12
.../24 dev enp1s0 proto kernel scope link src 46.105.121.211
.../24 dev wgclient_home scope link

LXC INSTANCE

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:fa:70:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::216:3eff:fefa:70ab/64 scope link
valid_lft forever preferred_lft forever

ip r
nothing

sudo ps aux | grep dnsmasq
root 7438 0.0 0.0 13144 1096 pts/0 S+ 11:17 0:00 grep --color=auto dnsmasq

it’s not started?

ss -ulpn
    State     Recv-Q      Send-Q            Local Address:Port            Peer Address:Port                                                                                                                                                                                                        
    UNCONN    0           0                       0.0.0.0:58406                0.0.0.0:*                                                                                                                                                                                                           
    UNCONN    0           0                     127.0.0.1:53                   0.0.0.0:*         users:(("named",pid=1921,fd=518),("named",pid=1921,fd=517),("named",pid=1921,fd=516),("named",pid=1921,fd=515),("named",pid=1921,fd=514),("named",pid=1921,fd=513),("named",pid=1921,fd=512))     
    UNCONN    0           0                 127.0.0.53%lo:53                   0.0.0.0:*         users:(("systemd-resolve",pid=1647,fd=12))                                                                                                                                                        
    UNCONN    0           0                          [::]:58406                   [::]:*                                                                                                                                                                                                           
    UNCONN    0           0                         [::1]:53                      [::]:*         users:(("named",pid=1921,fd=525),("named",pid=1921,fd=524),("named",pid=1921,fd=523),("named",pid=1921,fd=522),("named",pid=1921,fd=521),("named",pid=1921,fd=520),("named",pid=1921,fd=519))

grep dnsmasq /var/snap/lxd/common/lxd/logs/lxd.log
t=2021-06-17T10:50:21+0200 lvl=warn msg=“Skipping AppArmor for dnsmasq due to raw.dnsmasq being set” driver=bridge name=lxdbr0 network=lxdbr0 project=default
t=2021-06-17T10:50:22+0200 lvl=eror msg=“The dnsmasq process exited prematurely” driver=bridge err=“Process exited with non-zero value 1” network=lxdbr0 project=default

Looks like the same as dnsmask process exited prematurely · Issue #8905 · lxc/lxd · GitHub potentially. This seems to be an issue with the recent change to core20 in the snap. See LXD snap transitioning to core20 and losing i386 support

I don’t understand, the server it’s not an arch386. at the moment there is no solution or workaround to gain connectivity again?

The move to core20 affected all architectures, but meant that i386 isn’t supported any more.

I’m currently looking into if there is a workaround, like we did for the lvm tools.

ok, sorry

It seems to be an issue with snap on bionic hosts.

I have this is issue also: ubuntu host 18.04 and containers have no IP addresses this morning.

I’m tracking this over at

I reopened the original LVM issue as it would be good to try and figure out what was going on there:

Conclusion is that for anyone who’s using nested LVM (VG in a LV), the change of LVM version that occurred by LXD moving from a core18 to a core20 base changed LVM’s behavior to not scanning such LVs. As having LVM perform this scan every time is quite resource intensive, we don’t want to alter the default LVM configuration to do it.

Instead anyone depending on such a setup should do snap set lxd lvm.external=true followed by systemctl reload snap.lxd.daemon. This will reconfigure the snap to rely on your system’s LVM tools and configuration rather than the snap’s.
This will work seamlessly with the only requirement being that you must have the LVM tools installed on your system.

1 Like