Containers fail to start after server upgrade to Ubuntu 18.04.2 LTS

harolds · February 26, 2019, 3:09pm

After doing an upgrade of Ubuntu to 18.04.2 TLS and upgrading all applications LXD does not startup anymore.

root@dev ~ # snap list
Name  Version    Rev    Tracking  Publisher   Notes
core  16-2.37.2  6405   stable    canonical✓  core
lxd   3.10       10234  stable    canonical✓  -

Running lxd in debug mode show this:

root@dev ~ # lxd --debug --group lxd
INFO[02-26|15:56:28] LXD 3.10 is starting in normal mode      path=/var/snap/lxd/common/lxd
INFO[02-26|15:56:28] Kernel uid/gid map:
INFO[02-26|15:56:28]  - u 0 0 4294967295
INFO[02-26|15:56:28]  - g 0 0 4294967295
INFO[02-26|15:56:28] Configured LXD uid/gid map:
INFO[02-26|15:56:28]  - u 0 100000 65536
INFO[02-26|15:56:28]  - g 0 100000 65536
WARN[02-26|15:56:28] CGroup memory swap accounting is disabled, swap limits will be ignored.
INFO[02-26|15:56:28] Kernel features:
INFO[02-26|15:56:28]  - netnsid-based network retrieval: no
INFO[02-26|15:56:28]  - uevent injection: no
INFO[02-26|15:56:28]  - unprivileged file capabilities: yes
INFO[02-26|15:56:28] Initializing local database
DBUG[02-26|15:56:28] Initializing database gateway
DBUG[02-26|15:56:28] Start database node                      id=1 address=
DBUG[02-26|15:56:29] Raft: Restored from snapshot 1-1093162-1551171358762
DBUG[02-26|15:56:29] Raft: Initial configuration (index=1): [{Suffrage:Voter ID:1 Address:0}]
DBUG[02-26|15:56:29] Raft: Node at 0 [Leader] entering Leader state
DBUG[02-26|15:56:29] Dqlite: starting event loop
DBUG[02-26|15:56:29] Dqlite: accepting connections
INFO[02-26|15:56:29] Starting /dev/lxd handler:
INFO[02-26|15:56:29]  - binding devlxd socket                 socket=/var/snap/lxd/common/lxd/devlxd/sock
INFO[02-26|15:56:29] REST API daemon:
INFO[02-26|15:56:29]  - binding Unix socket                   socket=/var/snap/lxd/common/lxd/unix.socket
INFO[02-26|15:56:29]  - binding TCP socket                    socket=[::]:8443
INFO[02-26|15:56:29] Initializing global database
DBUG[02-26|15:56:29] Dqlite: handling new connection (fd=20)
DBUG[02-26|15:56:29] Dqlite: connected address=0 attempt=0
INFO[02-26|15:56:29] Initializing storage pools
DBUG[02-26|15:56:29] Initializing and checking storage pool "tank"
DBUG[02-26|15:56:29] Checking ZFS storage pool "tank"
INFO[02-26|15:56:29] Initializing networks
DBUG[02-26|15:56:29] Connecting to a remote simplestreams server
DBUG[02-26|15:56:29] New task operation: 4bc10f85-6a67-4c9a-a05b-cc3e9a4eabf8
INFO[02-26|15:56:29] Pruning leftover image files
DBUG[02-26|15:56:29] Started task operation: 4bc10f85-6a67-4c9a-a05b-cc3e9a4eabf8
INFO[02-26|15:56:29] Done pruning leftover image files
INFO[02-26|15:56:29] Loading daemon configuration
DBUG[02-26|15:56:29] Initialized inotify with file descriptor 19
DBUG[02-26|15:56:29] Success for task operation: 4bc10f85-6a67-4c9a-a05b-cc3e9a4eabf8
DBUG[02-26|15:56:29] New task operation: 8e394aea-968d-485f-a7a1-98425e8eca97
INFO[02-26|15:56:29] Pruning expired images
DBUG[02-26|15:56:29] Started task operation: 8e394aea-968d-485f-a7a1-98425e8eca97
INFO[02-26|15:56:29] Done pruning expired images
DBUG[02-26|15:56:29] New task operation: 10d0f446-1877-49b5-ad63-fdbee7af736a
INFO[02-26|15:56:29] Pruning expired container backups
DBUG[02-26|15:56:29] Started task operation: 10d0f446-1877-49b5-ad63-fdbee7af736a
INFO[02-26|15:56:29] Done pruning expired container backups
DBUG[02-26|15:56:29] New task operation: e5c8ac8e-7465-4ac6-9e64-642b0d3f4b59
INFO[02-26|15:56:29] Expiring log files
DBUG[02-26|15:56:29] Started task operation: e5c8ac8e-7465-4ac6-9e64-642b0d3f4b59
INFO[02-26|15:56:29] Done expiring log files
DBUG[02-26|15:56:29] Success for task operation: 8e394aea-968d-485f-a7a1-98425e8eca97
DBUG[02-26|15:56:29] New task operation: 47255fb4-9e00-43ba-9578-7cfa6a3ce128
INFO[02-26|15:56:29] Updating instance types
DBUG[02-26|15:56:29] Started task operation: 47255fb4-9e00-43ba-9578-7cfa6a3ce128
INFO[02-26|15:56:29] Done updating instance types
DBUG[02-26|15:56:29] Success for task operation: e5c8ac8e-7465-4ac6-9e64-642b0d3f4b59
EROR[02-26|15:56:29] Failed to start container 'kasact': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasact2': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasdev': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasdev2': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasdev3': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasdev4': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasdev5': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasdev6': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasdev7': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasdev8': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasedsntest': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kastest': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kastest2': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasupg2': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasupg3': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasupg4': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasupg5': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasupg6': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'kasupg7': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
EROR[02-26|15:56:29] Failed to start container 'ubuntu-1204-lts': Daemon failed to setup shared mounts base: no such file or directory.
Does security.nesting need to be turned on?
DBUG[02-26|15:56:29] New task operation: 06f15751-c6f2-44aa-8646-a9c20ff48c14
INFO[02-26|15:56:29] Updating images
DBUG[02-26|15:56:29] Started task operation: 06f15751-c6f2-44aa-8646-a9c20ff48c14
INFO[02-26|15:56:29] Done updating images
DBUG[02-26|15:56:29] Success for task operation: 10d0f446-1877-49b5-ad63-fdbee7af736a
DBUG[02-26|15:56:29] Database error: &errors.errorString{s:"sql: no rows in result set"}
DBUG[02-26|15:56:29] Database error: &errors.errorString{s:"sql: no rows in result set"}
EROR[02-26|15:56:29] Error getting source image               err="No such object" fp=4659e616fe0c2f2b706e28097104719760a7f3f1b05610c7ac1bdcfa6af716a2
DBUG[02-26|15:56:29] Database error: &errors.errorString{s:"sql: no rows in result set"}
DBUG[02-26|15:56:29] Success for task operation: 06f15751-c6f2-44aa-8646-a9c20ff48c14
DBUG[02-26|15:56:31] Success for task operation: 47255fb4-9e00-43ba-9578-7cfa6a3ce128

We are not running containers inside containers. Just simpel setup with number of Ubuntu containers.

root@dev ~ # lxc info --show-log kasupg4
Name: kasupg4
Location: none
Remote: unix://
Architecture: x86_64
Created: 2019/02/25 08:27 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc kasupg4 20190226094650.489 WARN     conf - conf.c:lxc_setup_devpts:1641 - Invalid argument - Failed to unmount old devpts instance

stgraber · February 26, 2019, 3:24pm

Ok,

Please start LXD normally with systemctl start snap.lxd.daemon snap.lxd.daemon.unix.socket
Then confirm that you can talk to it with lxc info.

At that point, please show:

sudo nsenter -t $(pgrep daemon.start) -m – ls -lh /var/snap/lxd/common/lxd
sudo nsenter -t $(pgrep daemon.start) -m – cat /proc/self/mountinfo
sudo journalctl -u snap.lxd.daemon -n 300

harolds · February 26, 2019, 3:26pm

Hmm…

root@dev ~ # systemctl start snap.lxd.daemon snap.lxd.daemon.unix.socket
Failed to start snap.lxd.daemon.service: Unit snap.lxd.daemon.service not found.
Job for snap.lxd.daemon.unix.socket failed.
See “systemctl status snap.lxd.daemon.unix.socket” and “journalctl -xe” for details.

stgraber · February 26, 2019, 3:43pm

Special, what’s going on with that system, can you show:

systemctl -a | grep snap.lxd

harolds · February 26, 2019, 3:44pm

root@dev ~ # systemctl -a | grep snap.lxd
  snap-lxd-10206.mount                                                                                                                                    loaded    active   mounted   Mount unit for lxd, revision 10206                       
  snap-lxd-10218.mount                                                                                                                                    loaded    active   mounted   Mount unit for lxd, revision 10218                       
  snap-lxd-10234.mount                                                                                                                                    loaded    active   mounted   Mount unit for lxd, revision 10234                       
● snap.lxd.daemon.service                                                                                                                                 not-found inactive dead      snap.lxd.daemon.service                                  
  snap.lxd.daemon.unix.socket                                                                                                                             loaded    inactive dead      Socket unix for snap application lxd.daemon

stgraber · February 26, 2019, 3:45pm

What does snap info lxd show you?

harolds · February 26, 2019, 3:45pm

root@dev ~ # snap info lxd
name:      lxd
summary:   System container manager and API
publisher: Canonical✓
contact:   https://github.com/lxc/lxd/issues
license:   unset
description: |
  **LXD is a system container manager**

  With LXD you can run hundreds of containers of a variety of Linux
  distributions, apply resource limits, pass in directories, USB devices
  or GPUs and setup any network and storage you want.

  LXD containers are lightweight, secure by default and a great
  alternative to running Linux virtual machines.


  **Run any Linux distribution you want**

  Pre-made images are available for Ubuntu, Alpine Linux, ArchLinux,
  CentOS, Debian, Fedora, Gentoo, OpenSUSE and more.

  A full list of available images can be [found here](https://images.linuxcontainers.org)

  Can't find the distribution you want? It's easy to make your own images too, either using our
  `distrobuilder` tool or by assembling your own image tarball by hand.


  **Containers at scale**

  LXD is network aware and all interactions go through a simple REST API,
  making it possible to remotely interact with containers on remote
  systems, copying and moving them as you wish.

  Want to go big? LXD also has built-in clustering support,
  letting you turn dozens of servers into one big LXD server.


  **Configuration options**

  Supported options for the LXD snap (`snap set lxd KEY=VALUE`):
   - criu.enable: Enable experimental live-migration support [default=false]
   - daemon.debug: Increases logging to debug level [default=false]
   - daemon.group: Group of users that can interact with LXD [default=lxd]
   - ceph.builtin: Use snap-specific ceph configuration [default=false]
   - openvswitch.builtin: Run a snap-specific OVS daemon [default=false]

  [Documentation](https://lxd.readthedocs.io)
snap-id:      J60k4JY0HppjwOjW8dZdYc8obXKxujRu
tracking:     stable
refresh-date: yesterday at 22:46 CET
channels:
  stable:        3.10        2019-02-25 (10234) 54MB -
  candidate:     3.10        2019-02-25 (10234) 54MB -
  beta:          ↑
  edge:          git-0d70553 2019-02-26 (10250) 54MB -
  3.0/stable:    3.0.3       2018-11-26  (9663) 53MB -
  3.0/candidate: 3.0.3       2019-01-19  (9942) 53MB -
  3.0/beta:      ↑
  3.0/edge:      git-eaa62ce 2019-02-19 (10212) 53MB -
  2.0/stable:    2.0.11      2018-07-30  (8023) 28MB -
  2.0/candidate: 2.0.11      2018-07-27  (8023) 28MB -
  2.0/beta:      ↑
  2.0/edge:      git-c7c4cc8 2018-10-19  (9257) 26MB -
installed:       3.10                   (10234) 54MB -

stgraber · February 26, 2019, 3:46pm

Hmm, okay, weird, can you try:

snap refresh lxd --edge
snap refresh lxd --stable

Maybe that will motivate snapd to restore the missing systemd unit.

harolds · February 26, 2019, 3:49pm

Wow that fixed it!

Super that you respond so quickly to issues and help fix it. I love LXD

stgraber · February 26, 2019, 3:50pm

Good that this worked, though it’s a bit concerning that snapd could just forget to generate a systemd unit like that…

harolds · February 26, 2019, 3:52pm

Upgrading to newer Ubuntu versions is always a bit tricky. Lots of warnings/etc. Hope this helps others too.

stgraber · February 26, 2019, 3:53pm

Could you show snap changes? This may contain useful information.

harolds · February 26, 2019, 3:53pm

root@dev ~ # snap changes
ID   Status  Spawn                   Ready                   Summary
109  Done    yesterday at 22:45 CET  yesterday at 22:46 CET  Auto-refresh snap "lxd"
110  Done    today at 14:36 CET      today at 14:36 CET      Refresh all snaps: no updates
111  Error   today at 14:39 CET      today at 14:39 CET      Running service command
112  Done    today at 15:22 CET      today at 15:22 CET      Refresh all snaps: no updates
113  Done    today at 15:31 CET      today at 15:31 CET      Switch "core" snap to stable
114  Done    today at 15:31 CET      today at 15:32 CET      Refresh snap "core"
115  Error   today at 15:34 CET      today at 15:34 CET      Running service command
116  Done    today at 15:34 CET      today at 15:34 CET      Refresh all snaps: no updates
117  Done    today at 16:47 CET      today at 16:47 CET      Refresh "lxd" snap from "edge" channel
118  Done    today at 16:47 CET      today at 16:47 CET      Refresh "lxd" snap

harolds · February 26, 2019, 3:54pm

snap was on beta core, I thought that that might be an issue so switched back to stable.

stgraber · February 26, 2019, 3:55pm

Can you show snap tasks 115 and snap tasks 111?

harolds · February 26, 2019, 3:57pm

root@dev ~ # snap tasks 115
Status  Spawn               Ready               Summary
Error   today at 15:34 CET  today at 15:34 CET  restart of [lxd.activate lxd.daemon]

......................................................................
restart of [lxd.activate lxd.daemon]

2019-02-26T15:34:34+01:00 ERROR # systemctl restart snap.lxd.activate.service snap.lxd.daemon.service
Failed to restart snap.lxd.activate.service: Unit snap.lxd.activate.service not found.
Failed to restart snap.lxd.daemon.service: Unit snap.lxd.daemon.service not found.

2019-02-26T15:34:34+01:00 ERROR exit status 5

root@dev ~ # snap tasks 111
Status  Spawn               Ready               Summary
Error   today at 14:39 CET  today at 14:39 CET  restart of [lxd.activate lxd.daemon]

......................................................................
restart of [lxd.activate lxd.daemon]

2019-02-26T14:39:59+01:00 ERROR # systemctl restart snap.lxd.activate.service snap.lxd.daemon.service
Failed to restart snap.lxd.activate.service: Unit snap.lxd.activate.service not found.
Failed to restart snap.lxd.daemon.service: Unit snap.lxd.daemon.service not found.

2019-02-26T14:39:59+01:00 ERROR exit status 5

stgraber · February 26, 2019, 4:02pm

Thanks, I forwarded that to the snapd team

harolds · February 26, 2019, 4:05pm

Cool! Thank you too.