Issues on boot and shutdown of host machine after upgrading from 3.0 deb to 3.5 snap

LXD
1

Hi

I am using Xubuntu 18.04 and upgraded LXD from 3.0 deb to 3.5 snap. I did it by installing the snap and then using lxd.migrate. No issues happened with the containers but on next boot the host machine started to present some problematic behavior.

On boot some systemd service/socket (didn’t see a target failing) may fail to start. This means a LXD socket may fail or the network may be down on boot, requiring to manually start the service. One example where the system became without internet is:

[   10.788076] systemd[1]: network-online.target: Found ordering cycle on NetworkManager-wait-online.service/start
[   10.788080] systemd[1]: network-online.target: Found dependency on NetworkManager.service/start
[   10.788082] systemd[1]: network-online.target: Found dependency on basic.target/start
[   10.788084] systemd[1]: network-online.target: Found dependency on sockets.target/start
[   10.788087] systemd[1]: network-online.target: Found dependency on snap.lxd.daemon.unix.socket/start
[   10.788089] systemd[1]: network-online.target: Found dependency on network-online.target/start
[   10.788092] systemd[1]: network-online.target: Job NetworkManager-wait-online.service/start deleted to break ordering cycle starting with network-online.target/start

Shutdown became very slow after installing the snap. With the 3.0 deb, around 20 seconds were required to stop all running containers and then shutdown would happen, but with the snap the LXD containers “fail to stop” (the containers stop properly but the service not), the snap fails to unmount and shutdown requires 5 minutes. I tried stopping the containers before starting the shutdown but the long wait still happens.

Boot issues seems to happen 100% of times, with the service being disabled different on every boot, shutdown issues happens often but not always. I would say 80% of the times.

I noticed this happening with 4.15.0-36, 4.17.19 and 4.19-rc6 kernel versions.

2

If you google on the above message, you will find several bug reports.
For your case, may I dare to ask whether you are not using NetworkManager but opted instead for something else? It looks like NetworkManager-wait-online.service would finish up after NetworkManager.service is completed.

3

While there are multiple results about that particular message, at least to me none of the search results seem to be related to LXD. It’s important to say I had no issues with systemd before installing the LXD snap.

I chose no alternate network service to use, I’m using the default Xubuntu 18.04 desktop settings.

The systemd messages at dmesg aren’t always the same. Sometimes there are only warning and no errors, as on this boot, where it shows:

[   10.541871] systemd[1]: basic.target: Found ordering cycle on sockets.target/start
[   10.541876] systemd[1]: basic.target: Found dependency on snap.lxd.daemon.unix.socket/start
[   10.541878] systemd[1]: basic.target: Found dependency on network-online.target/start
[   10.541880] systemd[1]: basic.target: Found dependency on network.target/start
[   10.541882] systemd[1]: basic.target: Found dependency on NetworkManager.service/start
[   10.541884] systemd[1]: basic.target: Found dependency on dbus.service/start
[   10.541886] systemd[1]: basic.target: Found dependency on basic.target/start

Apparently it was able to fix all services by itself this time, but sometimes one systemd service may be deleted to solve the problem, as in the first message happened with NetworkManager-wait-online.service. It’s not always the same service that is deleted.

4

The .deb LXD isn’t still installed is it ?

5

They are no longer installed. It was removed after using lxd.migrate command, I suppose it’s one of the steps of the migration.

I noticed that disabling the systemd socket snap.lxd.daemon.unix.socket made the boot issues disappear. I’ll be sincere and say I don’t know what I am losing by disabling this socket, but at least my system is booting correctly with no systemd warning or errors. I can say this is a workaround for now for the boot issues.

I still don’t know what should I do to correct the shutdown issue, which seems to be a problem in the unmount of the snap.

6

That’s the upstream report of this issue, looks like a fix is on its way. I’d recommend to keep the discussion going over there instead.

7

The problem on boot was corrected but on shutdown the problem still persists. Shutdown is taking more than 5 minutes. If I use lxd shutdown and then shutdown the system, it is as fast as before I upgraded from deb to snap.

LXD snap revision is now 9298.

8

We’ve had a few reports of longer than expected delay on shutdown, we’ll need to investigate.

Normally, lxd shutdown should take up to 30s to complete, that’s unless containers have been configured with a longer host shutdown timeout.

9

When I use lxd shutdown it takes around 20 seconds to complete. But if I shutdown the computer without using lxd shutdown previously it takes the 5 minutes.

It seems lxd shutdown is working correctly when I use it from a terminal, but on shutdown or it does not work or it is not called.

10

Thanks, that’s useful feedback. On shutdown, can you get to the systemd output (usually pressing ESC) and see exactly what it is it’s waiting for?

I wonder if it’s getting stuck on some mount units, effectively not knowing in what order to undo the mounts.

11

It’s not always the same thing. The following message always appear on system shutdown (the revision may be different):

[FAILED] Failed unmounting Mount Unit for lxd, revision 9298.

Often the 5 minutes wait happens after the last message, as:

Starting Reboot...

Another message that may appear before or after the Starting Shutdown or Starting Reboot is:

systemd-journald[PID]: Failed to send WATCHDOG=1 notification: Connection refused

Rarely on shutdown it may shows it’s waiting to stop the unit snap.lxd.daemon.service, but it stops after around 2 minutes but it waits for additional time until the system shutdown or reboot.

12

Can you show us:

  • lxc list
  • lxc storage list
  • cat /proc/self/mountinfo
  • cat /proc/$(pgrep daemon.start)/mountinfo

That should make it easier to reproduce the issue.

13

lxc list

$ lxc list
+-------------------+---------+------+-----------------------------------------------+------------+-----------+
|       NAME        |  STATE  | IPV4 |                     IPV6                      |    TYPE    | SNAPSHOTS |
+-------------------+---------+------+-----------------------------------------------+------------+-----------+
| apache-ubuntu1804 | RUNNING |      | fd42:7276:dc98:4040:216:3eff:feb7:946e (eth0) | PERSISTENT |           |
+-------------------+---------+------+-----------------------------------------------+------------+-----------+
| gitea             | RUNNING |      | fd42:7276:dc98:4040:216:3eff:fe1d:30ee (eth0) | PERSISTENT |           |
+-------------------+---------+------+-----------------------------------------------+------------+-----------+
| starbound         | STOPPED |      |                                               | PERSISTENT |           |
+-------------------+---------+------+-----------------------------------------------+------------+-----------+
| ubuntuamd64       | STOPPED |      |                                               | PERSISTENT |           |
+-------------------+---------+------+-----------------------------------------------+------------+-----------+
| ubuntui386        | STOPPED |      |                                               | PERSISTENT |           |
+-------------------+---------+------+-----------------------------------------------+------------+-----------+

lxc storage list

+---------+-------------+--------+------------------------------------------------+---------+
|  NAME   | DESCRIPTION | DRIVER |                     SOURCE                     | USED BY |
+---------+-------------+--------+------------------------------------------------+---------+
| default |             | dir    | /var/snap/lxd/common/lxd/storage-pools/default | 6       |
+---------+-------------+--------+------------------------------------------------+---------+

cat /proc/self/mountinfo

22 29 0:21 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw
23 29 0:4 / /proc rw,nosuid,nodev,noexec,relatime shared:14 - proc proc rw
24 29 0:6 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=3970292k,nr_inodes=992573,mode=755
25 24 0:22 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
26 29 0:23 / /run rw,nosuid,noexec,relatime shared:5 - tmpfs tmpfs rw,size=807192k,mode=755
29 0 8:11 / / rw,relatime shared:1 - ext4 /dev/sda11 rw,errors=remount-ro
30 22 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
31 24 0:26 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw
32 26 0:27 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k
33 22 0:28 / /sys/fs/cgroup ro,nosuid,nodev,noexec shared:9 - tmpfs tmpfs ro,mode=755
34 33 0:29 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime shared:10 - cgroup2 cgroup rw
35 33 0:30 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime shared:11 - cgroup cgroup rw,xattr,name=systemd
36 22 0:31 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:12 - pstore pstore rw
37 22 0:32 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:13 - efivarfs efivarfs rw
38 33 0:33 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime shared:15 - cgroup cgroup rw,perf_event
39 33 0:34 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime shared:16 - cgroup cgroup rw,blkio
40 33 0:35 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime shared:17 - cgroup cgroup rw,net_cls,net_prio
41 33 0:36 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime shared:18 - cgroup cgroup rw,hugetlb
42 33 0:37 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime shared:19 - cgroup cgroup rw,cpuset
43 33 0:38 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime shared:20 - cgroup cgroup rw,devices
44 33 0:39 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime shared:21 - cgroup cgroup rw,cpu,cpuacct
45 33 0:40 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime shared:22 - cgroup cgroup rw,memory
46 33 0:41 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime shared:23 - cgroup cgroup rw,pids
47 33 0:42 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime shared:24 - cgroup cgroup rw,rdma
48 33 0:43 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime shared:25 - cgroup cgroup rw,freezer
49 23 0:44 / /proc/sys/fs/binfmt_misc rw,relatime shared:26 - autofs systemd-1 rw,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19338
52 24 0:19 / /dev/mqueue rw,relatime shared:27 - mqueue mqueue rw
50 22 0:8 / /sys/kernel/debug rw,relatime shared:28 - debugfs debugfs rw
51 24 0:45 / /dev/hugepages rw,relatime shared:29 - hugetlbfs hugetlbfs rw,pagesize=2M
53 50 0:11 / /sys/kernel/debug/tracing rw,relatime shared:30 - tracefs tracefs rw
85 22 0:20 / /sys/kernel/config rw,relatime shared:31 - configfs configfs rw
87 22 0:46 / /sys/fs/fuse/connections rw,relatime shared:32 - fusectl fusectl rw
89 29 7:0 / /snap/core/5548 ro,nodev,relatime shared:33 - squashfs /dev/loop0 ro
91 29 7:2 / /snap/ponysay/2 ro,nodev,relatime shared:34 - squashfs /dev/loop2 ro
95 29 7:5 / /snap/micro/648 ro,nodev,relatime shared:35 - squashfs /dev/loop5 ro
96 29 7:7 / /snap/core/5328 ro,nodev,relatime shared:36 - squashfs /dev/loop7 ro
93 29 7:4 / /snap/lxd/9277 ro,nodev,relatime shared:37 - squashfs /dev/loop4 ro
90 29 7:1 / /snap/core/5662 ro,nodev,relatime shared:38 - squashfs /dev/loop1 ro
94 29 7:6 / /snap/lxd/9298 ro,nodev,relatime shared:39 - squashfs /dev/loop6 ro
105 29 0:47 / /tmp rw,relatime shared:41 - tmpfs tmpfs rw
107 105 0:48 / /tmp/dumps rw,relatime shared:42 - tmpfs tmpfs rw,size=98304k
109 29 8:3 / /mnt/W10 rw,relatime shared:43 - fuseblk /dev/sda3 rw,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096
111 29 8:1 / /boot/efi rw,relatime shared:44 - vfat /dev/sda1 rw,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
113 29 8:10 / /home rw,relatime shared:45 - ext4 /dev/sda10 rw,stripe=32735
115 49 0:49 / /proc/sys/fs/binfmt_misc rw,relatime shared:46 - binfmt_misc binfmt_misc rw
616 26 0:23 /snapd/ns /run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=807192k,mode=755
652 616 0:3 mnt:[4026532419] /run/snapd/ns/lxd.mnt rw - nsfs nsfs rw
812 29 0:57 / /var/snap/lxd/common/ns rw,relatime - tmpfs tmpfs rw,size=1024k,mode=700
827 812 0:3 mnt:[4026532420] /var/snap/lxd/common/ns/shmounts rw - nsfs nsfs rw
485 26 0:65 / /run/user/1000 rw,nosuid,nodev,relatime shared:302 - tmpfs tmpfs rw,size=807188k,mode=700,uid=1000,gid=1000
526 485 0:82 / /run/user/1000/gvfs rw,nosuid,nodev,relatime shared:354 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
462 29 7:8 / /snap/lxd/9354 ro,nodev,relatime shared:301 - squashfs /dev/loop8 ro

cat /proc/$(pgrep daemon.start)/mountinfo

628 747 8:11 / /var/lib/snapd/hostfs rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
634 628 0:23 / /var/lib/snapd/hostfs/run rw,nosuid,noexec,relatime master:5 - tmpfs tmpfs rw,size=807192k,mode=755
635 634 0:27 / /var/lib/snapd/hostfs/run/lock rw,nosuid,nodev,noexec,relatime master:6 - tmpfs tmpfs rw,size=5120k
636 634 0:23 /snapd/ns /var/lib/snapd/hostfs/run/snapd/ns rw,nosuid,noexec,relatime - tmpfs tmpfs rw,size=807192k,mode=755
662 628 7:0 / /var/lib/snapd/hostfs/snap/core/5548 ro,nodev,relatime master:33 - squashfs /dev/loop0 ro
663 628 7:2 / /var/lib/snapd/hostfs/snap/ponysay/2 ro,nodev,relatime master:34 - squashfs /dev/loop2 ro
664 628 7:5 / /var/lib/snapd/hostfs/snap/micro/648 ro,nodev,relatime master:35 - squashfs /dev/loop5 ro
665 628 7:7 / /var/lib/snapd/hostfs/snap/core/5328 ro,nodev,relatime master:36 - squashfs /dev/loop7 ro
666 628 7:4 / /var/lib/snapd/hostfs/snap/lxd/9277 ro,nodev,relatime master:37 - squashfs /dev/loop4 ro
667 628 7:1 / /var/lib/snapd/hostfs/snap/core/5662 ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
668 628 7:6 / /var/lib/snapd/hostfs/snap/lxd/9298 ro,nodev,relatime master:39 - squashfs /dev/loop6 ro
670 628 0:47 / /var/lib/snapd/hostfs/tmp rw,relatime master:41 - tmpfs tmpfs rw
671 670 0:48 / /var/lib/snapd/hostfs/tmp/dumps rw,relatime master:42 - tmpfs tmpfs rw,size=98304k
672 628 8:3 / /var/lib/snapd/hostfs/mnt/W10 rw,relatime master:43 - fuseblk /dev/sda3 rw,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096
673 628 8:1 / /var/lib/snapd/hostfs/boot/efi rw,relatime master:44 - vfat /dev/sda1 rw,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro
674 628 8:10 / /var/lib/snapd/hostfs/home rw,relatime master:45 - ext4 /dev/sda10 rw,stripe=32735
687 627 7:1 / / ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
688 687 0:6 / /dev rw,nosuid,relatime master:2 - devtmpfs udev rw,size=3970292k,nr_inodes=992573,mode=755
689 688 0:22 / /dev/pts rw,nosuid,noexec,relatime master:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
690 688 0:26 / /dev/shm rw,nosuid,nodev master:4 - tmpfs tmpfs rw
691 688 0:19 / /dev/mqueue rw,relatime master:27 - mqueue mqueue rw
692 688 0:45 / /dev/hugepages rw,relatime master:29 - hugetlbfs hugetlbfs rw,pagesize=2M
694 687 8:10 / /home rw,relatime master:45 - ext4 /dev/sda10 rw,stripe=32735
695 687 8:11 /root /root rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
696 687 0:4 / /proc rw,nosuid,nodev,noexec,relatime master:14 - proc proc rw
697 696 0:44 / /proc/sys/fs/binfmt_misc rw,relatime master:26 - autofs systemd-1 rw,fd=39,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=19338
698 697 0:49 / /proc/sys/fs/binfmt_misc rw,relatime master:46 - binfmt_misc binfmt_misc rw
699 687 0:21 / /sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
700 699 0:7 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime master:8 - securityfs securityfs rw
701 699 0:28 / /sys/fs/cgroup ro,nosuid,nodev,noexec master:9 - tmpfs tmpfs ro,mode=755
702 701 0:29 / /sys/fs/cgroup/unified rw,nosuid,nodev,noexec,relatime master:10 - cgroup2 cgroup rw
703 701 0:30 / /sys/fs/cgroup/systemd rw,nosuid,nodev,noexec,relatime master:11 - cgroup cgroup rw,xattr,name=systemd
704 701 0:33 / /sys/fs/cgroup/perf_event rw,nosuid,nodev,noexec,relatime master:15 - cgroup cgroup rw,perf_event
705 701 0:34 / /sys/fs/cgroup/blkio rw,nosuid,nodev,noexec,relatime master:16 - cgroup cgroup rw,blkio
706 701 0:35 / /sys/fs/cgroup/net_cls,net_prio rw,nosuid,nodev,noexec,relatime master:17 - cgroup cgroup rw,net_cls,net_prio
707 701 0:36 / /sys/fs/cgroup/hugetlb rw,nosuid,nodev,noexec,relatime master:18 - cgroup cgroup rw,hugetlb
708 701 0:37 / /sys/fs/cgroup/cpuset rw,nosuid,nodev,noexec,relatime master:19 - cgroup cgroup rw,cpuset
709 701 0:38 / /sys/fs/cgroup/devices rw,nosuid,nodev,noexec,relatime master:20 - cgroup cgroup rw,devices
710 701 0:39 / /sys/fs/cgroup/cpu,cpuacct rw,nosuid,nodev,noexec,relatime master:21 - cgroup cgroup rw,cpu,cpuacct
711 701 0:40 / /sys/fs/cgroup/memory rw,nosuid,nodev,noexec,relatime master:22 - cgroup cgroup rw,memory
712 701 0:41 / /sys/fs/cgroup/pids rw,nosuid,nodev,noexec,relatime master:23 - cgroup cgroup rw,pids
713 701 0:42 / /sys/fs/cgroup/rdma rw,nosuid,nodev,noexec,relatime master:24 - cgroup cgroup rw,rdma
714 701 0:43 / /sys/fs/cgroup/freezer rw,nosuid,nodev,noexec,relatime master:25 - cgroup cgroup rw,freezer
715 699 0:31 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime master:12 - pstore pstore rw
716 699 0:32 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime master:13 - efivarfs efivarfs rw
717 699 0:8 / /sys/kernel/debug rw,relatime master:28 - debugfs debugfs rw
718 717 0:11 / /sys/kernel/debug/tracing rw,relatime master:30 - tracefs tracefs rw
719 699 0:20 / /sys/kernel/config rw,relatime master:31 - configfs configfs rw
720 699 0:46 / /sys/fs/fuse/connections rw,relatime master:32 - fusectl fusectl rw
721 687 0:47 / /tmp rw,relatime master:41 - tmpfs tmpfs rw
722 721 0:48 / /tmp/dumps rw,relatime master:42 - tmpfs tmpfs rw,size=98304k
723 687 8:11 /var/snap /var/snap rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
724 632 8:11 /var/lib/snapd /var/lib/snapd rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
725 687 8:11 /var/tmp /var/tmp rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
729 687 8:11 /lib/modules /lib/modules rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
730 687 8:11 /usr/src /usr/src rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
731 687 8:11 /var/log /var/log rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
732 687 8:11 /media /media rw,relatime shared:1 - ext4 /dev/sda11 rw,errors=remount-ro
734 687 8:11 /mnt /mnt rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
735 734 8:3 / /mnt/W10 rw,relatime master:43 - fuseblk /dev/sda3 rw,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096
738 687 8:11 /snap /snap rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
739 738 7:0 / /snap/core/5548 ro,nodev,relatime master:33 - squashfs /dev/loop0 ro
740 738 7:2 / /snap/ponysay/2 ro,nodev,relatime master:34 - squashfs /dev/loop2 ro
741 738 7:5 / /snap/micro/648 ro,nodev,relatime master:35 - squashfs /dev/loop5 ro
742 738 7:7 / /snap/core/5328 ro,nodev,relatime master:36 - squashfs /dev/loop7 ro
743 738 7:4 / /snap/lxd/9277 ro,nodev,relatime master:37 - squashfs /dev/loop4 ro
744 738 7:1 / /snap/core/5662 ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
745 738 7:6 / /snap/lxd/9298 ro,nodev,relatime master:39 - squashfs /dev/loop6 ro
747 724 8:11 /var/lib/snapd/hostfs /var/lib/snapd/hostfs rw,relatime - ext4 /dev/sda11 rw,errors=remount-ro
629 721 0:47 /snap.0_lxd_1oaaJO/tmp /tmp rw,relatime - tmpfs tmpfs rw
630 689 0:54 / /dev/pts rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
631 688 0:54 /ptmx /dev/ptmx rw,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
632 687 0:55 / /var/lib rw,nosuid,nodev,relatime - tmpfs none rw
633 632 7:1 /var/lib/apparmor /var/lib/apparmor ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
637 632 7:1 /var/lib/classic /var/lib/classic ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
638 632 7:1 /var/lib/cloud /var/lib/cloud ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
639 632 7:1 /var/lib/console-conf /var/lib/console-conf ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
640 632 7:1 /var/lib/dbus /var/lib/dbus ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
641 632 7:1 /var/lib/dhcp /var/lib/dhcp ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
642 632 7:1 /var/lib/extrausers /var/lib/extrausers ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
643 632 7:1 /var/lib/initramfs-tools /var/lib/initramfs-tools ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
644 632 7:1 /var/lib/initscripts /var/lib/initscripts ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
645 632 7:1 /var/lib/insserv /var/lib/insserv ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
646 632 7:1 /var/lib/logrotate /var/lib/logrotate ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
647 632 7:1 /var/lib/machines /var/lib/machines ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
648 632 7:1 /var/lib/misc /var/lib/misc ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
649 632 7:1 /var/lib/pam /var/lib/pam ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
650 632 7:1 /var/lib/python /var/lib/python ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
651 632 7:1 /var/lib/resolvconf /var/lib/resolvconf ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
653 632 7:1 /var/lib/sudo /var/lib/sudo ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
654 632 7:1 /var/lib/systemd /var/lib/systemd ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
655 632 7:1 /var/lib/ubuntu-fan /var/lib/ubuntu-fan ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
656 632 7:1 /var/lib/ucf /var/lib/ucf ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
657 632 7:1 /var/lib/update-rc.d /var/lib/update-rc.d ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
658 632 7:1 /var/lib/urandom /var/lib/urandom ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
659 632 7:1 /var/lib/vim /var/lib/vim ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
660 632 7:1 /var/lib/waagent /var/lib/waagent ro,nodev,relatime master:38 - squashfs /dev/loop1 ro
816 723 0:57 / /var/snap/lxd/common/ns rw,relatime master:469 - tmpfs tmpfs rw,size=1024k,mode=700
815 628 0:57 / /var/lib/snapd/hostfs/var/snap/lxd/common/ns rw,relatime master:469 - tmpfs tmpfs rw,size=1024k,mode=700
828 723 0:58 / /var/snap/lxd/common/shmounts rw,relatime shared:477 - tmpfs tmpfs rw,size=1024k,mode=711
798 687 7:6 /wrappers/kmod /bin/kmod ro,nodev,relatime master:39 - squashfs /dev/loop6 ro
475 628 0:6 / /var/lib/snapd/hostfs/dev rw,nosuid,relatime master:2 - devtmpfs udev rw,size=3970292k,nr_inodes=992573,mode=755
476 628 0:4 / /var/lib/snapd/hostfs/proc rw,nosuid,nodev,noexec,relatime master:14 - proc proc rw
477 628 0:21 / /var/lib/snapd/hostfs/sys rw,nosuid,nodev,noexec,relatime master:7 - sysfs sysfs rw
478 687 8:11 /usr/share/ca-certificates /usr/share/ca-certificates rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
480 828 0:62 / /var/snap/lxd/common/shmounts/lxcfs rw,nosuid,nodev,relatime shared:300 - fuse.lxcfs lxcfs rw,user_id=0,group_id=0,allow_other
487 634 0:65 / /var/lib/snapd/hostfs/run/user/1000 rw,nosuid,nodev,relatime master:302 - tmpfs tmpfs rw,size=807188k,mode=700,uid=1000,gid=1000
686 487 0:82 / /var/lib/snapd/hostfs/run/user/1000/gvfs rw,nosuid,nodev,relatime master:354 - fuse.gvfsd-fuse gvfsd-fuse rw,user_id=1000,group_id=1000
482 738 7:8 / /snap/lxd/9354 ro,nodev,relatime master:301 - squashfs /dev/loop8 ro
464 628 7:8 / /var/lib/snapd/hostfs/snap/lxd/9354 ro,nodev,relatime master:301 - squashfs /dev/loop8 ro
483 798 7:8 /wrappers/kmod /bin/kmod ro,nodev,relatime master:301 - squashfs /dev/loop8 ro
92 483 7:8 /wrappers/kmod /bin/kmod ro,nodev,relatime master:301 - squashfs /dev/loop8 ro
104 687 8:11 /boot /boot rw,relatime master:1 - ext4 /dev/sda11 rw,errors=remount-ro
158 687 0:53 / /run rw,nosuid,nodev,relatime - tmpfs tmpfs rw,mode=755
204 687 0:56 / /etc rw,relatime - tmpfs tmpfs rw,mode=755
254 828 0:59 / /var/snap/lxd/common/shmounts/containers rw,relatime shared:40 - tmpfs tmpfs rw,size=100k,mode=711
382 723 0:60 / /var/snap/lxd/common/lxd/devlxd rw,relatime - tmpfs tmpfs rw,size=100k,mode=755