OK cool, well that’s interesting that that allowed them to start.
Can you show the output of sudo lvs
again now, and also can you lxc shell <instance>
ok into one of the containers?
LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
containers_gdg pool_0 -wi-ao---k 3.00g
containers_git pool_0 -wi-ao---k 3.00g
containers_iceCast pool_0 -wi-ao---k 3.00g
containers_m--clamav pool_0 -wi-ao---k 3.00g
containers_m--clamavOLD pool_0 -wi------k 3.00g
containers_m--dcc pool_0 -wi------k 3.00g
containers_m--dcot pool_0 -wi-ao---k 3.00g
containers_m--pfix pool_0 -wi-ao---k 3.00g
containers_m--spam18 pool_0 -wi-ao---k 3.00g
containers_mysql pool_0 -wi-ao---k 3.00g
containers_named pool_0 -wi-ao---k 3.00g
containers_owncloud pool_0 -wi-ao---k 3.00g
containers_rCube pool_0 -wi-ao---k 3.00g
containers_rCube14 pool_0 -wi------k 3.00g
containers_redis pool_0 -wi-ao---k 3.00g
containers_solr pool_0 -wi-ao---k 3.00g
containers_www--ext pool_0 -wi------k 3.00g
containers_www--std pool_0 -wi-ao---k 3.00g
containers_wwwHurdega pool_0 -wi-ao---k 3.00g
custom_default_git_data pool_0 -wi-ao---k 3.00g
custom_default_data pool_OC -wi-ao---k 193.00g
custom_default_solrData pool_data -wi-ao---k <11.18g
custom_default_vmail_data pool_data -wi-ao---k <29.32g
custom_default_www_audio pool_data -wi-ao---k 5.00g
home vg -wi-ao---- <19.53g
lxc_pool00 vg -wi-ao---- 60.00g
lxc_pool01 vg -wi-ao---- 20.00g
lxc_poolOC00 vg -wi-ao---- 220.00g
lxc_pooldata00 vg -wi-ao---- 50.00g
lxc_pooldata01 vg -wi-a----- 50.00g
lxc shell works!
now it’s just the network the problem…
Can you show output of lxc config show <instance> --expanded
for one of the problem instances.
Then also show lxc network show <network>
for the network that the instance is connected to.
Also can you show the output of ip a
and ip r
from the LXD host and inside one of the problem instances.
ips are redacted…
architecture: x86_64
config:
boot.autostart: "true"
boot.autostart.priority: "1000"
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20190212.1)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: "20190212.1"
image.version: "18.04"
volatile.base_image: 35f6bff57c250208c6dc73445eefe8c1020853a1bc8571ebd8bfb1fe3a239504
volatile.eth0.host_name: veth3515a489
volatile.eth0.hwaddr: 00:16:3e:fa:70:ab
volatile.idmap.base: "0"
volatile.idmap.current: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.idmap.next: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.idmap: '[{"Isuid":true,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
volatile.last_state.power: RUNNING
volatile.uuid: f5e3b6f5-7059-4995-999e-4c5841dfceee
devices:
bind_4_tcp:
bind: host
connect: tcp:10.4.43.176:53
listen: tcp:##.##.##.##:53
nat: "true"
type: proxy
bind_4_udp:
bind: host
connect: udp:10.4.43.176:53
listen: udp:##.##.##.##:53
nat: "true"
type: proxy
bind_6_tcp:
bind: host
connect: tcp:[fd42:26ae:9c2f:e23:216:3eff:fefa:70ab]:53
listen: tcp:[##:##:#:#::#]:53
nat: "true"
type: proxy
bind_6_udp:
bind: host
connect: udp:[fd42:26ae:9c2f:e23:216:3eff:fefa:70ab]:53
listen: udp:[##:##:#:#::#]:53
nat: "true"
type: proxy
eth0:
ipv4.address: 10.4.43.176
ipv6.address: fd42:26ae:9c2f:e23:216:3eff:fefa:70ab
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: pool0
type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""
lxc network show lxdbr0
config:
ipv4.address: 10.4.43.1/24
ipv4.nat: “true”
ipv6.address: fd42:26ae:9c2f:e23::1/64
ipv6.nat: “true”
raw.dnsmasq: |
auth-zone=lxd
dns-loop-detect
description: “”
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/gdg
- /1.0/instances/git
- /1.0/instances/iceCast
- /1.0/instances/m-clamav
- /1.0/instances/m-clamavOLD
- /1.0/instances/m-dcc
- /1.0/instances/m-dcot
- /1.0/instances/m-pfix
- /1.0/instances/m-spam18
- /1.0/instances/mysql
- /1.0/instances/named
- /1.0/instances/owncloud
- /1.0/instances/rCube
- /1.0/instances/rCube14
- /1.0/instances/redis
- /1.0/instances/solr
- /1.0/instances/www-ext
- /1.0/instances/www-std
- /1.0/instances/wwwHurdega
- /1.0/profiles/default
managed: true
status: Created
locations:
- none
Cool thanks. And what does sudo ps aux | grep dnsmasq
show?
Also sudo ss -ulpn
?
Can you also show sudo grep /var/snap/lxd/common/lxd/logs/lxd.log dnsmasq
please
LXC HOST
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP group default qlen 1000
link/ether 00:25:90:79:85:4a brd ff:ff:ff:ff:ff:ff
inet 46.105.121.211/24 brd 46.105.121.255 scope global enp1s0
valid_lft forever preferred_lft forever
inet6 2001:41d0:2:e8d3::1/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::225:90ff:fe79:854a/64 scope link
valid_lft forever preferred_lft forever
3: enp2s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:25:90:79:85:4b brd ff:ff:ff:ff:ff:ff
4: wgclient_home: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 10.127.10.12/24 scope global wgclient_home
valid_lft forever preferred_lft forever
5: lxdbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:97:44:03 brd ff:ff:ff:ff:ff:ff
inet 10.4.43.1/24 scope global lxdbr0
valid_lft forever preferred_lft forever
inet6 fd42:26ae:9c2f:e23::1/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe97:4403/64 scope link
valid_lft forever preferred_lft forever
7: vethae7df84d@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 5a:72:a1:27:14:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
9: veth3515a489@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether ea:4d:8a:2c:91:14 brd ff:ff:ff:ff:ff:ff link-netnsid 1
11: veth11bb0b31@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 02:00:0e:6e:61:f2 brd ff:ff:ff:ff:ff:ff link-netnsid 2
13: veth58fde0b5@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether e6:30:b7:ae:2f:e0 brd ff:ff:ff:ff:ff:ff link-netnsid 3
15: veth982cdf44@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 36:82:ef:bb:05:c1 brd ff:ff:ff:ff:ff:ff link-netnsid 4
17: veth712d0c5b@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 4e:6b:12:f1:96:56 brd ff:ff:ff:ff:ff:ff link-netnsid 5
19: veth36f55459@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether fa:d4:b5:f3:55:52 brd ff:ff:ff:ff:ff:ff link-netnsid 6
21: veth4bd7e789@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether c6:bb:34:17:72:56 brd ff:ff:ff:ff:ff:ff link-netnsid 7
23: veth97903699@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 9a:1a:2f:66:9d:de brd ff:ff:ff:ff:ff:ff link-netnsid 8
25: vethcb7e14a3@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether a2:a2:a7:da:18:13 brd ff:ff:ff:ff:ff:ff link-netnsid 9
27: vethb07621eb@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 46:aa:78:14:c8:58 brd ff:ff:ff:ff:ff:ff link-netnsid 10
29: vethd6fe5eba@if28: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 16:59:31:6a:1b:ca brd ff:ff:ff:ff:ff:ff link-netnsid 11
31: vetheb147d01@if30: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 32:d5:14:1c:65:71 brd ff:ff:ff:ff:ff:ff link-netnsid 12
33: veth4c1f23ac@if32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether d6:1c:fa:c9:c1:2d brd ff:ff:ff:ff:ff:ff link-netnsid 13
35: vethfe8aae29@if34: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lxdbr0 state UP group default qlen 1000
link/ether 9e:16:f1:80:1a:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 14
ip r
default via ... dev enp1s0 proto static
10.4.43.0/24 dev lxdbr0 proto kernel scope link src 10.4.43.1
10.127.10.0/24 dev wgclient_home proto kernel scope link src 10.127.10.12
.../24 dev enp1s0 proto kernel scope link src 46.105.121.211
.../24 dev wgclient_home scope link
LXC INSTANCE
ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
8: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:fa:70:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::216:3eff:fefa:70ab/64 scope link
valid_lft forever preferred_lft forever
ip r
nothing
sudo ps aux | grep dnsmasq
root 7438 0.0 0.0 13144 1096 pts/0 S+ 11:17 0:00 grep --color=auto dnsmasq
it’s not started?
ss -ulpn
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 0.0.0.0:58406 0.0.0.0:*
UNCONN 0 0 127.0.0.1:53 0.0.0.0:* users:(("named",pid=1921,fd=518),("named",pid=1921,fd=517),("named",pid=1921,fd=516),("named",pid=1921,fd=515),("named",pid=1921,fd=514),("named",pid=1921,fd=513),("named",pid=1921,fd=512))
UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=1647,fd=12))
UNCONN 0 0 [::]:58406 [::]:*
UNCONN 0 0 [::1]:53 [::]:* users:(("named",pid=1921,fd=525),("named",pid=1921,fd=524),("named",pid=1921,fd=523),("named",pid=1921,fd=522),("named",pid=1921,fd=521),("named",pid=1921,fd=520),("named",pid=1921,fd=519))
grep dnsmasq /var/snap/lxd/common/lxd/logs/lxd.log
t=2021-06-17T10:50:21+0200 lvl=warn msg=“Skipping AppArmor for dnsmasq due to raw.dnsmasq being set” driver=bridge name=lxdbr0 network=lxdbr0 project=default
t=2021-06-17T10:50:22+0200 lvl=eror msg=“The dnsmasq process exited prematurely” driver=bridge err=“Process exited with non-zero value 1” network=lxdbr0 project=default
Looks like the same as dnsmask process exited prematurely · Issue #8905 · lxc/lxd · GitHub potentially. This seems to be an issue with the recent change to core20 in the snap. See LXD snap transitioning to core20 and losing i386 support
I don’t understand, the server it’s not an arch386. at the moment there is no solution or workaround to gain connectivity again?
The move to core20 affected all architectures, but meant that i386 isn’t supported any more.
I’m currently looking into if there is a workaround, like we did for the lvm tools.
ok, sorry
It seems to be an issue with snap on bionic hosts.
I have this is issue also: ubuntu host 18.04 and containers have no IP addresses this morning.
I’m tracking this over at
I reopened the original LVM issue as it would be good to try and figure out what was going on there:
Conclusion is that for anyone who’s using nested LVM (VG in a LV), the change of LVM version that occurred by LXD moving from a core18 to a core20 base changed LVM’s behavior to not scanning such LVs. As having LVM perform this scan every time is quite resource intensive, we don’t want to alter the default LVM configuration to do it.
Instead anyone depending on such a setup should do snap set lxd lvm.external=true
followed by systemctl reload snap.lxd.daemon
. This will reconfigure the snap to rely on your system’s LVM tools and configuration rather than the snap’s.
This will work seamlessly with the only requirement being that you must have the LVM tools installed on your system.