Containers starting/stopping on host reboot (recently moved to snap)

serbik · January 21, 2018, 10:42pm

Hi guys,

I am not able to keep my containers running after a host reboot. Some of them continually restart in a loop and others not starting at all.

Noticed some NIC errors in dmesg, can anyone help with this?

[ 1295.670459] br0: port 4(vethEVVR6K) entered forwarding state
[ 1295.718268] eth0: renamed from vethNQ6MTU                [ 1295.733469] IPv6: ADDRCONF(NETDEV_CHANGE): vethEVVR6K: link becomes ready                                            [ 1296.046930] device veth57DW12 entered promiscuous mode   [ 1296.047071] IPv6: ADDRCONF(NETDEV_UP): veth57DW12: link is not ready                                                 [ 1296.094273] eth0: renamed from veth4BX5HQ                [ 1296.117386] IPv6: ADDRCONF(NETDEV_CHANGE): veth57DW12: link becomes ready                                            [ 1296.117404] br0: port 5(veth57DW12) entered forwarding state                                                         [ 1296.117411] br0: port 5(veth57DW12) entered forwarding state                                                         [ 1296.495810] device veth3REOCE entered promiscuous mode   [ 1296.496069] IPv6: ADDRCONF(NETDEV_UP): veth3REOCE: link is not ready
[ 1296.496074] br0: port 6(veth3REOCE) entered forwarding state                                                         [ 1296.496080] br0: port 6(veth3REOCE) entered forwarding state                                                         [ 1296.542761] eth0: renamed from veth45A4WV                [ 1296.573497] IPv6: ADDRCONF(NETDEV_CHANGE): veth3REOCE: link becomes ready
[ 1297.000227] device vethIF5K7H entered promiscuous mode   [ 1297.002169] IPv6: ADDRCONF(NETDEV_UP): vethIF5K7H: link is not ready
[ 1297.002176] br0: port 7(vethIF5K7H) entered forwarding state
[ 1297.002182] br0: port 7(vethIF5K7H) entered forwarding state                                                         [ 1297.041026] eth0: renamed from vethQI4380                [ 1297.065203] br0: port 7(vethIF5K7H) entered disabled state
[ 1297.065627] IPv6: ADDRCONF(NETDEV_CHANGE): vethIF5K7H: link becomes ready                                            [ 1297.065644] br0: port 7(vethIF5K7H) entered forwarding state                                                         [ 1297.065651] br0: port 7(vethIF5K7H) entered forwarding state                                                         [ 1297.663844] br0: port 4(vethEVVR6K) entered disabled state                                                           [ 1297.731281] br0: port 3(vethX248GJ) entered disabled state
[ 1297.731405] device vethX248GJ left promiscuous mode      [ 1297.731415] br0: port 3(vethX248GJ) entered disabled state                                                           [ 1297.920010] br0: port 5(veth57DW12) entered disabled state
[ 1297.920108] device veth57DW12 left promiscuous mode      [ 1297.920109] br0: port 5(veth57DW12) entered disabled state                                                           [ 1297.954137] br0: port 4(vethEVVR6K) entered disabled state                                                           [ 1297.954209] device vethEVVR6K left promiscuous mode      [ 1297.954211] br0: port 4(vethEVVR6K) entered disabled state                                                           [ 1298.177976] br0: port 2(vethL9NJ65) entered disabled state                                                           [ 1298.214505] br0: port 7(vethIF5K7H) entered disabled state                                                           [ 1298.214593] device vethIF5K7H left promiscuous mode      [ 1298.214596] br0: port 7(vethIF5K7H) entered disabled state                                                           [ 1298.254746] br0: port 2(vethL9NJ65) entered disabled state                                                           [ 1298.254921] device vethL9NJ65 left promiscuous mode      [ 1298.254923] br0: port 2(vethL9NJ65) entered disabled state                                                           [ 1311.507619] br0: port 6(veth3REOCE) entered forwarding state                                                         [ 1327.287729] br0: port 6(veth3REOCE) entered disabled state
[ 1327.288066] device veth3REOCE left promiscuous mode      [ 1327.288075] br0: port 6(veth3REOCE) entered disabled state

serbik · January 21, 2018, 10:51pm

Failed to open file '/sys/class/net/vethDQDA6O/operstate': No such file or directory

lxd.daemon[30139]: lvl=warn msg="Unable to update backup.yaml at this time." name=mumble

serbik · January 22, 2018, 8:12am

Appears fixed after editing /etc/lxc/default.conf to the new network bridge

However now when I reboot the host, none of the containers automatically start even although they are running at shutdown and also set to persistent?

stgraber · January 28, 2018, 8:58pm

The containers not coming back online is odd, what does /var/snap/lxd/common/lxd/logs/lxd.log contain immediately after a reboot?

I wonder if LXD maybe started before your bridge was online, causing the containers to fail to start.

serbik · January 29, 2018, 12:49pm

Hi @stgraber

Thanks for your reply. Funnily enough I done a reboot there and all went well, all containers auto started.
I wonder if it was the lxd bridge issue you mentioned, as since I disabled this at /etc/default/ it seems to be resolved.

Here’s output of lxd.log for this reboot (I manually restarted the transmission container due to unrelated network issue):

 lvl=info msg="LXD 2.21 is starting in normal mode" path=/var/snap/lxd/common/lxd t=2018-01-29T12:37:54+0000
lvl=info msg="Kernel uid/gid map:" t=2018-01-29T12:37:54+0000
lvl=info msg=" - u 0 0 4294967295" t=2018-01-29T12:37:54+0000
lvl=info msg=" - g 0 0 4294967295" t=2018-01-29T12:37:54+0000
lvl=info msg="Configured LXD uid/gid map:" t=2018-01-29T12:37:54+0000
lvl=info msg=" - u 0 1000000 1000000000" t=2018-01-29T12:37:54+0000
lvl=info msg=" - g 0 1000000 1000000000" t=2018-01-29T12:37:54+0000
lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." t=2018-01-29T12:37:54+0000
lvl=info msg="LXD isn't socket activated" t=2018-01-29T12:37:55+0000
lvl=info msg="Starting /dev/lxd handler:" t=2018-01-29T12:37:55+0000
lvl=info msg=" - binding devlxd socket" socket=/var/snap/lxd/common/lxd/devlxd/sock t=2018-01-29T12:37:55+0000
lvl=info msg="REST API daemon:" t=2018-01-29T12:37:55+0000
lvl=info msg=" - binding Unix socket" socket=/var/snap/lxd/common/lxd/unix.socket t=2018-01-29T12:37:55+0000
lvl=info msg=" - binding TCP socket" socket=[::]:8443 t=2018-01-29T12:37:55+0000
lvl=info msg="Pruning expired images" t=2018-01-29T12:37:55+0000
lvl=info msg="Done pruning expired images" t=2018-01-29T12:37:55+0000
lvl=info msg="Expiring log files" t=2018-01-29T12:37:55+0000
lvl=info msg="Updating instance types" t=2018-01-29T12:37:55+0000
lvl=info msg="Done expiring log files" t=2018-01-29T12:37:55+0000
lvl=info msg="Updating images" t=2018-01-29T12:37:55+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=checkmk t=2018-01-29T12:37:55+0000
action=start created=2018-01-20T01:28:24+0000 ephemeral=false lvl=info msg="Starting container" name=checkmk stateful=false t=2018-01-29T12:37:55+0000 used=2018-01-24T22:07:35+0000
action=start created=2018-01-20T01:28:24+0000 ephemeral=false lvl=info msg="Started container" name=checkmk stateful=false t=2018-01-29T12:37:56+0000 used=2018-01-24T22:07:35+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=gs-test t=2018-01-29T12:37:56+0000
lvl=info msg="Done updating images" t=2018-01-29T12:37:56+0000
action=start created=2018-01-19T20:44:42+0000 ephemeral=false lvl=info msg="Starting container" name=gs-test stateful=false t=2018-01-29T12:37:56+0000 used=2018-01-24T22:16:59+0000
lvl=info msg="Done updating instance types" t=2018-01-29T12:37:57+0000
action=start created=2018-01-19T20:44:42+0000 ephemeral=false lvl=info msg="Started container" name=gs-test stateful=false t=2018-01-29T12:37:57+0000 used=2018-01-24T22:16:59+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=mta t=2018-01-29T12:37:57+0000
action=start created=2018-01-19T02:07:41+0000 ephemeral=false lvl=info msg="Starting container" name=mta stateful=false t=2018-01-29T12:37:58+0000 used=2018-01-24T22:16:40+0000
action=start created=2018-01-19T02:07:41+0000 ephemeral=false lvl=info msg="Started container" name=mta stateful=false t=2018-01-29T12:37:59+0000 used=2018-01-24T22:16:40+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=mumble t=2018-01-29T12:37:59+0000
action=start created=2018-01-20T16:08:01+0000 ephemeral=false lvl=info msg="Starting container" name=mumble stateful=false t=2018-01-29T12:37:59+0000 used=2018-01-24T22:16:34+0000
action=start created=2018-01-20T16:08:01+0000 ephemeral=false lvl=info msg="Started container" name=mumble stateful=false t=2018-01-29T12:38:00+0000 used=2018-01-24T22:16:34+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=nextcloud t=2018-01-29T12:38:00+0000
action=start created=2018-01-17T00:58:48+0000 ephemeral=false lvl=info msg="Starting container" name=nextcloud stateful=false t=2018-01-29T12:38:01+0000 used=2018-01-24T22:17:09+0000
action=start created=2018-01-17T00:58:48+0000 ephemeral=false lvl=info msg="Started container" name=nextcloud stateful=false t=2018-01-29T12:38:02+0000 used=2018-01-24T22:17:09+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=pihole t=2018-01-29T12:38:02+0000
action=start created=2018-01-19T20:42:15+0000 ephemeral=false lvl=info msg="Starting container" name=pihole stateful=false t=2018-01-29T12:38:03+0000 used=2018-01-24T22:16:04+0000
action=start created=2018-01-19T20:42:15+0000 ephemeral=false lvl=info msg="Started container" name=pihole stateful=false t=2018-01-29T12:38:04+0000 used=2018-01-24T22:16:04+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=plex t=2018-01-29T12:38:04+0000
action=start created=2018-01-17T00:59:41+0000 ephemeral=false lvl=info msg="Starting container" name=plex stateful=false t=2018-01-29T12:38:05+0000 used=2018-01-24T22:16:15+0000
action=start created=2018-01-17T00:59:41+0000 ephemeral=false lvl=info msg="Started container" name=plex stateful=false t=2018-01-29T12:38:05+0000 used=2018-01-24T22:16:15+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=transmission t=2018-01-29T12:38:05+0000
action=start created=2018-01-19T23:57:30+0000 ephemeral=false lvl=info msg="Starting container" name=transmission stateful=false t=2018-01-29T12:38:06+0000 used=2018-01-24T22:16:23+0000
action=start created=2018-01-19T23:57:30+0000 ephemeral=false lvl=info msg="Started container" name=transmission stateful=false t=2018-01-29T12:38:07+0000 used=2018-01-24T22:16:23+0000
action=shutdown created=2018-01-19T23:57:30+0000 ephemeral=false lvl=info msg="Shutting down container" name=transmission t=2018-01-29T12:39:31+0000 timeout=-1s used=2018-01-29T12:38:06+0000
action=shutdown created=2018-01-19T23:57:30+0000 ephemeral=false lvl=info msg="Shut down container" name=transmission t=2018-01-29T12:39:34+0000 timeout=-1s used=2018-01-29T12:38:06+0000
lvl=warn msg="Unable to update backup.yaml at this time." name=transmission t=2018-01-29T12:39:34+0000
action=start created=2018-01-19T23:57:30+0000 ephemeral=false lvl=info msg="Starting container" name=transmission stateful=false t=2018-01-29T12:39:34+0000 used=2018-01-29T12:38:06+0000
action=start created=2018-01-19T23:57:30+0000 ephemeral=false lvl=info msg="Started container" name=transmission stateful=false t=2018-01-29T12:39:34+0000 used=2018-01-29T12:38:06+0000

ernstae · January 31, 2018, 12:19am

I’m not sure if this will solve the problem you’re having, but I’ve added this line of configuration to my default (and other) profiles:

config:
  boot.autostart: true

You should be able to set this for all your existing containers at the container config level with

$ lxc config set containername boot.autostart true

The TL;DR would be something more like this:

$ for containername in $(lxc list -c n --format csv) ; 
  do 
    lxc config set $containername boot.autostart true ; 
  done;

Hope this helps!

Andrew