Sorry for going straight to github issues in the past - I didn’t know about this forum. I’ll suss this one out here and file an issue if it is in fact an issue.
root@wyzsrv:~# cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
source /etc/network/interfaces.d/*
# The loopback network interface
auto lo
iface lo inet loopback
root@wyzsrv:~# cat /etc/network/interfaces.d/br0
auto br0
iface br0 inet dhcp
bridge_ports enp6s7
iface br0 inet6 auto
iface enp6s7 inet manual
iface enp6s7 inet6 manual
root@wyzsrv:~# lxc profile show default
config: {}
description: Default LXD profile
devices:
eth0:
nictype: bridged
parent: br0
type: nic
root:
path: /
pool: default
type: disk
name: default
used_by: []
root@wyzsrv:~# lxc image info ubuntu:lts
Fingerprint: 61d54418874f2f84e24ddd6934b3bb759ca76cbc49820da7d34f8b5b778e4816
Size: 156.15MB
Architecture: x86_64
Public: yes
Timestamps:
Created: 2017/10/11 00:00 UTC
Uploaded: 2017/10/11 00:00 UTC
Expires: 2021/04/21 00:00 UTC
Last used: never
Properties:
release: xenial
version: 16.04
architecture: amd64
label: release
serial: 20171011
description: ubuntu 16.04 LTS amd64 (release) (20171011)
os: ubuntu
Aliases:
- 16.04
- 16.04/amd64
- default
- default/amd64
- lts
- lts/amd64
- x
- x/amd64
- xenial
- xenial/amd64
Cached: no
Auto update: disabled
root@wyzsrv:~# lxc launch ubuntu:lts test
Creating test
Starting test
That’s the host background
tl;dr: metal running lxd 2.18 with btrfs can’t run nested stock lxd 2.0.10 on ubunto:lts (16.04)
scenario ran successfully with metal running 2.17
scenario: lxd init --auto et. al. inside of a stock container (ubuntu:lts) where the host runs btrfs.
error info:
root@wyzsrv:~# lxc exec test bash
root@test:~# ls /var/lib/lxd
root@test:~# ls /var/lib/lxd
root@test:~# ls /var/lib/lxd
root@test:~# ls /var/lib/lxd
unix.socket
root@test:~# systemctl status lxd.socket
● lxd.socket - LXD - unix socket
Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
Active: active (listening) since Fri 2017-10-20 02:31:33 UTC; 20s ago
Docs: man:lxd(1)
Listen: /var/lib/lxd/unix.socket (Stream)
Oct 20 02:31:33 test systemd[1]: Starting LXD - unix socket.
Oct 20 02:31:33 test systemd[1]: Listening on LXD - unix socket.
root@test:~# systemctl status lxd-bridge
● lxd-bridge.service - LXD - network bridge
Loaded: loaded (/lib/systemd/system/lxd-bridge.service; static; vendor preset: enabled)
Active: inactive (dead)
Docs: man:lxd(1)
root@test:~# systemctl status lxd
● lxd.service - LXD - main daemon
Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
Active: inactive (dead)
Docs: man:lxd(1)
root@test:~# lxd init --auto
error: Unable to talk to LXD: Get http://unix.socket/1.0: read unix @->/var/lib/lxd/unix.socket: read: connection reset by peer
root@test:~# systemctl status lxd.socket
● lxd.socket - LXD - unix socket
Loaded: loaded (/lib/systemd/system/lxd.socket; enabled; vendor preset: enabled)
Active: failed (Result: trigger-limit-hit) since Fri 2017-10-20 02:32:11 UTC; 3s ago
Docs: man:lxd(1)
Listen: /var/lib/lxd/unix.socket (Stream)
Oct 20 02:31:33 test systemd[1]: Starting LXD - unix socket.
Oct 20 02:31:33 test systemd[1]: Listening on LXD - unix socket.
root@test:~# systemctl status lxd-bridge
● lxd-bridge.service - LXD - network bridge
Loaded: loaded (/lib/systemd/system/lxd-bridge.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Fri 2017-10-20 02:32:07 UTC; 10s ago
Docs: man:lxd(1)
Process: 621 ExecStart=/usr/lib/lxd/lxd-bridge.start (code=exited, status=2)
Main PID: 621 (code=exited, status=2)
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
Oct 20 02:32:10 test systemd[1]: Failed to start LXD - network bridge.
root@test:~# systemctl status lxd
● lxd.service - LXD - main daemon
Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
Active: inactive (dead)
Docs: man:lxd(1)
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
Oct 20 02:32:10 test systemd[1]: lxd.service: Job lxd.service/start failed with result 'dependency'.
So again, the regression is that when running btrfs, you can’t nest 2.0.10 inside of a 2.18 host. My example didn’t show on the launch command -c security.nesting=true or `-c security.privileged=true’ but same results.
I don’t know if this happened on 2.17 but…
root@test:~# dmesg | grep 'BTRFS'
[ 17.968929] BTRFS: device label default devid 1 transid 132 /dev/dm-4
[ 2887.916342] BTRFS info (device dm-4): disk space caching is enabled
[ 2887.916352] BTRFS: has skinny extents
[ 2942.432341] BTRFS error (device dm-4): could not find root 8
[ 3079.764178] BTRFS error (device dm-4): could not find root 8
[ 3212.802244] BTRFS error (device dm-4): could not find root 8
[ 3413.799527] BTRFS error (device dm-4): could not find root 8
[ 4660.605285] BTRFS error (device dm-4): could not find root 8
[ 6408.103605] BTRFS error (device dm-4): could not find root 8
root@test:~#```
the same results with the same grep on the host
knowing that some btrfs changes came through 2.18, I dropped and recreated my storage volume. No difference.
This forum needs a 'preview' button so that i can see that I make sense lol, so I'm submitting and I'll edit if I don't
root@test:~# /usr/lib/lxd/lxd-bridge start
Bad argument `'
Try `iptables -h' or 'iptables --help' for more information.
Failed to setup lxd-bridge.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0-97-generic
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0-97-generic
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0-97-generic
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0-97-generic
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0-97-generic
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.4.0-97-generic
iptables v1.6.0: can't initialize iptables table `mangle': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip6_tables not found in directory /lib/modules/4.4.0-97-generic
ip6tables v1.6.0: can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip6_tables not found in directory /lib/modules/4.4.0-97-generic
ip6tables v1.6.0: can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip6_tables not found in directory /lib/modules/4.4.0-97-generic
ip6tables v1.6.0: can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip6_tables not found in directory /lib/modules/4.4.0-97-generic
ip6tables v1.6.0: can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-97-generic/modules.dep.bin'
modprobe: FATAL: Module ip6_tables not found in directory /lib/modules/4.4.0-97-generic
ip6tables v1.6.0: can't initialize ip6tables table `filter': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
root@test:~#
hmm… ok i had this in mind to say but it got lost,
rather than type your command i did, and then got:
root@test:~# lxc stop test
Generating a client certificate. This may take a minute...
If this is your first time using LXD, you should also run: sudo lxd init
To start your first container, try: lxc launch ubuntu:16.04
LXD socket not found; is LXD installed and running?
root@test:~#```
thinking that I'd shut the container down, issue your change, and restart
what I had in mind was time based - the above errors that I originally linked as a result of `lxd init --auto` would also happen if I never did `lxd init` but waited many minutes and then did that sequence of `systemctl status` command... the same would be true
You usually see such behavior if the network on the host is macvlan or bridged as in those case LXD on the host didn’t cause those modules to be loaded.
LXD socket not found; is LXD installed and running?
root@test:~# lxc stop test
LXD socket not found; is LXD installed and running?
root@test:~# lxc info
LXD socket not found; is LXD installed and running?
root@test:~# exit
root@wyzsrv:~# lxc stop test
root@wyzsrv:~# lxc start test
root@wyzsrv:~# lxc exec test bash
root@test:~# exit
root@wyzsrv:~# lxc stop test
root@wyzsrv:~# lxc config set test linux.kernel_modules ip_tables,ip6_tables
root@wyzsrv:~# lxc start test
root@wyzsrv:~# lxc exec test bash
root@test:~# lxd init --auto
LXD has been successfully configured.
root@test:~#
ok like a rockstar (as usual) you found the root of it all xD… so now I assert my original test case that I’ve been using:
In 2.17 I could create that container, and then create a sub-container - no issues
in 2.18 why would I suddenly need to include lxc config set test linux.kernel_modules ip_tables,ip6_tables (regression point)
2.17 command sequence (with btrfs set up as default storage): (replication steps)
lxc launch ubuntu:lts test
lxc exec test bash
lxd init --auto```
The above continues to work with my .travis.yml, & vagrant... it is only my 'sandbox' environment with BTRFS that fails. The others use the 'DIR' provider
repo: (for ruby'ists): https://github.com/NexusSW/lxd-common
There is no change in LXD 2.18 that would explain this.
The normal reason for this behavior is that your system would in the past have auto-loaded those iptables modules and isn’t anymore. The auto-loading can happen for any number of reason, running something as simple as “iptables -L” will cause them to auto-load.
On most LXD systems these modules get auto-loaded because the user selects our default “lxdbr0” bridge which does have IPv4 and IPv6 firewalling on it, causing the modules to get loaded.
It’s only on cases where you don’t have LXD run a bridge for you AND your system never attempts to read any of the iptables tables that those modules won’t be loaded and will require that linux.kernel_modules config key to ensure that they’re loaded prior to the container starting up.
i can’t find it, but I swear there was something titled approximately ‘add btfrs as a block file system’ in the 2.18 release - that’s what had me on alert about this release
Yeah, but that’s quite unrelated. This changelog item is about allowing users to create btrfs formatted filesystems when using a LVM or Ceph storage pool.
I’m honestly not sure it’s worth the trouble since we wouldn’t treat it as a regression anyway
Unprivileged containers can’t load kernel modules so if you depend on some specific modules being loaded (as is the case with nested LXD), then those should be listed in linux.kernel_modules or you should otherwise be sure that something will have loaded those modules prior to the container starting.
I get it now! Something didn’t quite sink in last night (I have to quit working so late)…
I rebooted my host to get it clean and repeated the above steps with lxc launch ubuntu:lts test -c security.privileged=true -c security.nesting=true that my tests do and I usually forget when I’m typing by hand (like I did above)
same results. Now with my (limited) understanding of module loading in a container, I believe with it being privileged it ‘could’ have loaded those modules(?), if it had those modules in its own filesystem. But it doesn’t so that explains the ‘same results’
At any rate, weird… I really couldn’t tell you what was loading those modules in 2.17 because my host is very stock and the only thing I’ve set up to use on it, beyond the basic desktop (that I don’t use - I ssh in), is that raw bridge, md, lvm, btrfs, all in support of lxd. Perhaps something else that updated alongside lxd when I did apt-get upgrade used to load them.
Meanwhile, I’m loading those modules in my default profile now. Thank you again!!!