[SOLVED] LXD 3.18 Stopped working: unix.socket: connect: connection refused

lepe1 · November 22, 2019, 8:59am

Ubuntu 18.04.1
Kernel: 4.15.0-34-generic
LXD: 3.18 rev. 12224 (stable)
Server running LXD since: 2018

Last week LXD stopped working suddenly (without any manual update of any kind). Automatic updates were ON (Ubuntu auto-update and snap refresh).

The error I was getting was that the database was locked everytime I wanted to do any operation with the containers. I executed: systemctl reload snap.lxd.daemon and all went back to normal.

However, today I had the same error again. I repeated the command, but this time all my containers (about 300) disconnected from the LXD network. Trying to fix that issue (restarting network, reverting snap, and other commands I can’t remember now), things got worst, and I ended up restarting the whole server.

Upon restarting all seems to be good but just as the last container went online again, I started to get Error: Get http://unix.socket/1.0: EOF error and something about a websocket.

I read that it could have been that the CPU was waiting for some process to finish, so I waited… Then every time I execute lxc list or any other lxc command, it shows:

Error: Get http://unix.socket/1.0: dial unix /var/snap/lxd/common/lxd/unix.socket: connect: connection refused

systemctl status snap.lxd.daemon.unix.socket reports:

● snap.lxd.daemon.unix.socket - Socket unix for snap application lxd.daemon
   Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.unix.socket; enabled; vendor preset: enabled)
   Active: failed (Result: service-start-limit-hit) since Fri 2019-11-22 17:55:39 JST; 15s ago
   Listen: /var/snap/lxd/common/lxd/unix.socket (Stream)

Nov 22 17:55:08 shiny.hidden.com systemd[1]: Listening on Socket unix for snap application lxd.daemon.
Nov 22 17:55:39 shiny.hidden.com systemd[1]: snap.lxd.daemon.unix.socket: Failed with result 'service-start-limit-hit'.

and systemctl status snap.lxd.daemon reports:

● snap.lxd.daemon.service - Service for snap application lxd.daemon
   Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-11-22 17:55:39 JST; 27s ago
  Process: 31597 ExecStart=/usr/bin/snap run lxd.daemon (code=exited, status=1/FAILURE)
 Main PID: 31597 (code=exited, status=1/FAILURE)

Nov 22 17:55:39 shiny.hidden.com systemd[1]: snap.lxd.daemon.service: Service hold-off time over, scheduling restart.
Nov 22 17:55:39 shiny.hidden.com systemd[1]: snap.lxd.daemon.service: Scheduled restart job, restart counter is at 8.
Nov 22 17:55:39 shiny.hidden.com systemd[1]: Stopped Service for snap application lxd.daemon.
Nov 22 17:55:39 shiny.hidden.com systemd[1]: snap.lxd.daemon.service: Start request repeated too quickly.
Nov 22 17:55:39 shiny.hidden.com systemd[1]: snap.lxd.daemon.service: Failed with result 'exit-code'.
Nov 22 17:55:39 shiny.hidden.com systemd[1]: Failed to start Service for snap application lxd.daemon.

I’m able to start such services, but they die after few seconds.
My containers are still running but I can not gain access to them. What can I do without having to restart the server or all the containers?

Thanks.

stgraber · November 22, 2019, 3:42pm

What do you get if you do:

systemctl stop snap.lxd.daemon.unix.socket
lxd --debug --group lxd

lepe1 · November 23, 2019, 2:42am

Crashing with Segmentation fault:

DBUG[11-23|11:41:44] Connecting to a local LXD over a Unix socket 
DBUG[11-23|11:41:44] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
INFO[11-23|11:41:44] LXD 3.18 is starting in normal mode      path=/var/snap/lxd/common/lxd
INFO[11-23|11:41:44] Kernel uid/gid map: 
INFO[11-23|11:41:44]  - u 0 0 4294967295 
INFO[11-23|11:41:44]  - g 0 0 4294967295 
INFO[11-23|11:41:44] Configured LXD uid/gid map: 
INFO[11-23|11:41:44]  - u 0 1000000 1000000000 
INFO[11-23|11:41:44]  - g 0 1000000 1000000000 
WARN[11-23|11:41:44] CGroup memory swap accounting is disabled, swap limits will be ignored. 
INFO[11-23|11:41:44] Kernel features: 
INFO[11-23|11:41:44]  - netnsid-based network retrieval: no 
INFO[11-23|11:41:44]  - uevent injection: no 
INFO[11-23|11:41:44]  - seccomp listener: no 
INFO[11-23|11:41:44]  - unprivileged file capabilities: yes 
INFO[11-23|11:41:44]  - shiftfs support: no 
INFO[11-23|11:41:44] Initializing local database 
DBUG[11-23|11:41:44] Initializing database gateway 
DBUG[11-23|11:41:44] Start database node                      id=1 address=
DBUG[11-23|11:41:45] Connecting to a local LXD over a Unix socket 
DBUG[11-23|11:41:45] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
DBUG[11-23|11:41:45] Detected stale unix socket, deleting 
DBUG[11-23|11:41:45] Detected stale unix socket, deleting 
INFO[11-23|11:41:45] Starting /dev/lxd handler: 
INFO[11-23|11:41:45]  - binding devlxd socket                 socket=/var/snap/lxd/common/lxd/devlxd/sock
INFO[11-23|11:41:45] REST API daemon: 
INFO[11-23|11:41:45]  - binding Unix socket                   socket=/var/snap/lxd/common/lxd/unix.socket
INFO[11-23|11:41:45]  - binding TCP socket                    socket=[::]:8443
INFO[11-23|11:41:45] Initializing global database 
DBUG[11-23|11:41:45] Dqlite: connected address=1 attempt=0 
Segmentation fault

The output of ps fauxww is: (some output was removed as it was repetitive and it exceeded the limits of this post. containers names were hidden)

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         2  0.0  0.0      0     0 ?        S    Nov22   0:17 [kthreadd]
root         4  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/0:0H]
root         6  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [mm_percpu_wq]
root         7  0.0  0.0      0     0 ?        S    Nov22   0:05  \_ [ksoftirqd/0]
root         8  0.3  0.0      0     0 ?        I    Nov22   3:40  \_ [rcu_sched]
root         9  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [rcu_bh]
root        10  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/0]
root        11  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/0]
root        12  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/0]
root        13  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/1]
root        14  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/1]
root        15  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/1]
root        16  0.0  0.0      0     0 ?        S    Nov22   0:05  \_ [ksoftirqd/1]
root        18  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/1:0H]
root        19  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/2]
root        20  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/2]
root        21  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/2]
root        22  0.0  0.0      0     0 ?        S    Nov22   0:04  \_ [ksoftirqd/2]
root        24  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/2:0H]
root        25  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/3]
root        26  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/3]
root        27  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/3]
root        28  0.0  0.0      0     0 ?        S    Nov22   0:05  \_ [ksoftirqd/3]
root        30  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/3:0H]
root        31  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/4]
root        32  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/4]
root        33  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/4]
root        34  0.0  0.0      0     0 ?        S    Nov22   0:05  \_ [ksoftirqd/4]
root        36  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/4:0H]
root        37  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/5]
root        38  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/5]
root        39  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/5]
root        40  0.0  0.0      0     0 ?        S    Nov22   0:05  \_ [ksoftirqd/5]
root        42  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/5:0H]
root        43  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/6]
root        44  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/6]
root        45  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/6]
root        46  0.0  0.0      0     0 ?        R    Nov22   0:04  \_ [ksoftirqd/6]
root        48  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/6:0H]
root        49  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/7]
root        50  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/7]
root        51  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/7]
root        52  0.0  0.0      0     0 ?        S    Nov22   0:04  \_ [ksoftirqd/7]
root        54  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/7:0H]
root        55  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/8]
root        56  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/8]
root        57  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/8]
root        58  0.0  0.0      0     0 ?        S    Nov22   0:04  \_ [ksoftirqd/8]
root        60  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/8:0H]
root        61  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/9]
root        62  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/9]
root        63  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/9]
root        64  0.0  0.0      0     0 ?        S    Nov22   0:03  \_ [ksoftirqd/9]
root        66  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/9:0H]
root        67  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/10]
root        68  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/10]
root        69  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/10]
root        70  0.0  0.0      0     0 ?        S    Nov22   0:04  \_ [ksoftirqd/10]
root        72  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/10:0H]
root        73  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/11]
root        74  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/11]
root        75  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/11]
root        76  0.0  0.0      0     0 ?        S    Nov22   0:03  \_ [ksoftirqd/11]
root        78  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/11:0H]
root        79  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/12]
root        80  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/12]
root        81  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/12]
root        82  0.0  0.0      0     0 ?        S    Nov22   0:04  \_ [ksoftirqd/12]
root        84  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/12:0H]
root        85  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/13]
root        86  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/13]
root        87  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/13]
root        88  0.0  0.0      0     0 ?        S    Nov22   0:03  \_ [ksoftirqd/13]
root        90  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/13:0H]
root        91  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/14]
root        92  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/14]
root        93  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/14]
root        94  0.0  0.0      0     0 ?        S    Nov22   0:04  \_ [ksoftirqd/14]
root        96  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/14:0H]
root        97  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [cpuhp/15]
root        98  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [watchdog/15]
root        99  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [migration/15]
root       100  0.0  0.0      0     0 ?        S    Nov22   0:02  \_ [ksoftirqd/15]
root       102  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kworker/15:0H]
root       103  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [kdevtmpfs]
root       104  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [netns]
root       105  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [rcu_tasks_kthre]
root       106  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [kauditd]
root       107  0.0  0.0      0     0 ?        I    Nov22   0:04  \_ [kworker/0:1]
root       110  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [khungtaskd]
root       111  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [oom_reaper]
root       112  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [writeback]
root       113  0.0  0.0      0     0 ?        S    Nov22   0:03  \_ [kcompactd0]
root       114  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [ksmd]
root       115  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [khugepaged]
root       116  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [crypto]
root       117  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kintegrityd]
root       118  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kblockd]
root       122  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [ata_sff]
root       123  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [md]
root       124  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [edac-poller]
root       125  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [devfreq_wq]
root       126  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [watchdogd]
root       129  3.2  0.0      0     0 ?        S    Nov22  39:28  \_ [kswapd0]
root       130  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [ecryptfs-kthrea]
root       172  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kthrotld]
root       173  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/3:1]
root       175  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/5:1]
root       179  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/9:1]
root       180  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/11:1]
root       181  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/12:1]
root       184  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [acpi_thermal_pm]
root       193  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [ipv6_addrconf]
root       202  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [kstrp]
root       220  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [charger_manager]
root       282  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [be_wq]
root       283  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [be_err_recover]
root       284  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_0]
root       285  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_0]
root       288  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_1]
root       289  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_1]
root       290  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_2]
root       291  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_2]
root       292  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_3]
root       293  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_3]
root       294  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_4]
root       295  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_4]
root       299  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_5]
root       300  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_5]
root       301  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_6]
root       302  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_6]
root       303  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_7]
root       304  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_7]
root       305  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_8]
root       306  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_8]
root       307  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_9]
root       308  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_9]
root       309  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [scsi_eh_10]
root       310  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [scsi_tmf_10]
root       323  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/3:1H]
root       324  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/4:1H]
root       334  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/11:1H]
root       338  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/13:1H]
root       341  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/12:1H]
root       345  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/6:1H]
root       347  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/7:1H]
root       348  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/8:1H]
root       350  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/2:1H]
root       351  0.0  0.0      0     0 ?        S    Nov22   0:28  \_ [jbd2/sda1-8]
root       352  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [ext4-rsv-conver]
root       354  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/1:1H]
root       359  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/15:1H]
root       364  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/0:1H]
root       368  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/10:1H]
root       382  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/9:1H]
root       386  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/5:1H]
root       391  5.7  0.0      0     0 ?        SN   Nov22  70:05  \_ [kipmi0]
root       439  0.0  0.0      0     0 ?        I<   Nov22   0:01  \_ [kworker/14:1H]
root       481  0.0  0.0      0     0 ?        S<   Nov22   0:01  \_ [loop0]
root       490  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [loop1]
root       519  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [loop2]
root       522  0.0  0.0      0     0 ?        S<   Nov22   0:01  \_ [loop3]
root       755  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [spl_system_task]
root       756  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [spl_delay_taskq]
root       757  0.0  0.0      0     0 ?        S<   Nov22   0:05  \_ [spl_dynamic_tas]
root       758  0.0  0.0      0     0 ?        S<   Nov22   0:03  \_ [spl_kmem_cache]
root       767  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [zvol]
root       773  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [arc_prune]
root       774  0.0  0.0      0     0 ?        S    Nov22   0:04  \_ [arc_reclaim]
root       775  0.0  0.0      0     0 ?        S    Nov22   0:07  \_ [dbu_evict]
root       776  0.0  0.0      0     0 ?        SN   Nov22   0:10  \_ [dbuf_evict]
root       779  0.0  0.0      0     0 ?        SN   Nov22   0:39  \_ [z_vdev_file]
root       780  0.0  0.0      0     0 ?        S    Nov22   0:02  \_ [l2arc_feed]
root       882  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_null_iss]
root       883  0.0  0.0      0     0 ?        S<   Nov22   0:13  \_ [z_null_int]
root       884  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_rd_iss]
root       885  0.1  0.0      0     0 ?        S<   Nov22   1:26  \_ [z_rd_int_0]
root       886  0.1  0.0      0     0 ?        S<   Nov22   1:14  \_ [z_rd_int_1]
root       887  0.1  0.0      0     0 ?        S<   Nov22   1:34  \_ [z_rd_int_2]
root       888  0.1  0.0      0     0 ?        S<   Nov22   1:27  \_ [z_rd_int_3]
root       889  0.1  0.0      0     0 ?        S<   Nov22   1:29  \_ [z_rd_int_4]
root       890  0.1  0.0      0     0 ?        S<   Nov22   1:19  \_ [z_rd_int_5]
root       891  0.1  0.0      0     0 ?        S<   Nov22   1:28  \_ [z_rd_int_6]
root       892  0.1  0.0      0     0 ?        S<   Nov22   1:33  \_ [z_rd_int_7]
root       893  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       894  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       895  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       896  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       897  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       898  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       899  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       900  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       901  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       902  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       903  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       904  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss]
root       905  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_iss_h]
root       906  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_0]
root       907  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_1]
root       908  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_2]
root       909  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_3]
root       910  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_4]
root       911  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_5]
root       912  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_6]
root       913  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_7]
root       914  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_wr_int_h]
root       915  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_iss_0]
root       916  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_iss_1]
root       917  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_iss_2]
root       918  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_iss_3]
root       919  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_iss_4]
root       920  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_iss_5]
root       921  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_iss_6]
root       922  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_iss_7]
root       923  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_fr_int]
root       924  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_cl_iss]
root       925  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_cl_int]
root       926  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_ioctl_iss]
root       927  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [z_ioctl_int]
root       928  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [z_zvol]
root       929  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [z_upgrade]
root       930  0.0  0.0      0     0 ?        S<   Nov22   0:00  \_ [metaslab_group_]
root       949  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       950  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       951  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       952  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       953  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       954  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       955  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       956  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       957  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       958  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       959  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       960  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_sync_taskq]
root       961  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       962  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       963  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       964  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       965  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       966  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       967  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       968  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       969  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       970  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       971  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       972  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       973  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       974  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       975  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       976  0.0  0.0      0     0 ?        SN   Nov22   0:00  \_ [dp_zil_clean_ta]
root       977  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [z_iput]
root      1536  0.0  0.0      0     0 ?        S    Nov22   0:00  \_ [txg_quiesce]
root      1537  0.0  0.0      0     0 ?        S    Nov22   0:01  \_ [txg_sync]
root      1538  0.0  0.0      0     0 ?        S    Nov22   0:02  \_ [mmp]
root      2964  0.0  0.0      0     0 ?        I<   Nov22   0:00  \_ [dio/sda1]
root     19943  0.0  0.0      0     0 ?        I    Nov22   0:03  \_ [kworker/1:2]
root     11213  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/10:0]
root     11260  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/2:0]
root     30846  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/15:2]
root     32454  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/6:1]
root     11944  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/14:0]
root     25111  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/0:0]
root     32460  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/6:0]
root     32464  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/13:2]
root     15834  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/4:2]
root     10165  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/12:0]
root     16463  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/3:0]
root     20563  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/5:2]
root     12730  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/4:1]
root     16006  0.0  0.0      0     0 ?        I    Nov22   0:02  \_ [kworker/13:5]
root     16538  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/1:0]
root     17782  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/10:1]
root     19403  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/11:0]
root     29544  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/15:1]
root      6635  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/7:2]
root      6738  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/9:0]
root      9208  0.0  0.0      0     0 ?        I    Nov22   0:00  \_ [kworker/8:2]
root     30285  0.0  0.0      0     0 ?        I    Nov22   0:01  \_ [kworker/8:0]
root     30286  0.0  0.0      0     0 ?        I    Nov22   0:01  \_ [kworker/7:0]
root     19273  0.0  0.0      0     0 ?        S<   01:44   0:00  \_ [z_rd_int_6]
root     19276  0.0  0.0      0     0 ?        S<   01:44   0:00  \_ [z_rd_int_5]
root      1013  0.0  0.0      0     0 ?        I    05:11   0:00  \_ [kworker/2:2]
root     18658  0.0  0.0      0     0 ?        I    06:47   0:00  \_ [kworker/14:2]
root      9450  0.0  0.0      0     0 ?        I    07:17   0:03  \_ [kworker/u32:0]
root      6994  0.0  0.0      0     0 ?        I    11:00   0:00  \_ [kworker/u32:1]
root     11605  0.0  0.0      0     0 ?        I    11:08   0:00  \_ [kworker/u32:2]
root      4379  0.0  0.0      0     0 ?        I    11:43   0:00  \_ [kworker/u32:3]
root         1  0.0  0.0 226464  8588 ?        Ss   Nov22   0:20 /sbin/init nosplash
root       394  0.0  0.1 143192 40816 ?        S<s  Nov22   0:07 /lib/systemd/systemd-journald
root       478  0.0  0.0  49300  3080 ?        Ss   Nov22   0:11 /lib/systemd/systemd-udevd
systemd+  1770  0.0  0.0 141912  2788 ?        Ssl  Nov22   0:00 /lib/systemd/systemd-timesyncd
systemd+  1772  0.0  0.0  70740  2800 ?        Ss   Nov22   0:00 /lib/systemd/systemd-resolved
root      1988  0.0  0.0 182972  2244 ?        Ssl  Nov22   0:00 /usr/sbin/zed -F
root      1989  0.0  0.0 110520  2784 ?        Ssl  Nov22   0:08 /usr/sbin/irqbalance --foreground
root      2000  0.0  0.0 167488  3700 ?        Ssl  Nov22   0:00 /usr/bin/python3 /usr/bin/networkd-dispatcher --run-startup-triggers
message+  2011  0.0  0.0  49928  3436 ?        Ss   Nov22   0:00 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only
root      2070  0.0  0.0 284632  2984 ?        Ssl  Nov22   0:02 /usr/lib/accountsservice/accounts-daemon
root      2082  0.0  0.0  28356  2596 ?        Ss   Nov22   0:01 /usr/sbin/cron -f
root      2083  0.0  0.0  70592  3852 ?        Ss   Nov22   0:00 /lib/systemd/systemd-logind
syslog    2087  0.0  0.0 263032  2352 ?        Ssl  Nov22   0:03 /usr/sbin/rsyslogd -n
root      2121  0.0  0.0  51684  4948 ?        Ss   Nov22   0:03 /usr/bin/perl -wT /usr/sbin/munin-node
daemon    2122  0.0  0.0  28332  2032 ?        Ss   Nov22   0:00 /usr/sbin/atd -f
root      2125  0.0  0.0   4520    16 ?        S<   Nov22   0:46 /usr/sbin/atopacctd
root      2130  0.0  0.0 2843712 10452 ?       Ssl  Nov22   0:21 /usr/lib/snapd/snapd
root      2165  0.0  0.0   6648  1684 ?        Ss   Nov22   0:00 /usr/sbin/rinetd
root      2196  0.2  0.0  43740  6280 ?        S    Nov22   2:35 /usr/bin/monit -c /etc/monit/monitrc
Debian-+  2252  4.0  0.0  63616  5496 ?        Ss   Nov22  49:14 /usr/sbin/snmpd -Lsd -Lf /dev/null -u Debian-snmp -g Debian-snmp -I -smux mteTrigger mteTriggerConf -f
root      2274  0.0  0.0  72296  2488 ?        Ss   Nov22   0:00 /usr/sbin/sshd -D
root      1974  0.0  0.0 103864  7104 ?        Ss   11:41   0:00  \_ sshd: ubuntu [priv]
ubuntu    2201  0.0  0.0 103864  3680 ?        S    11:41   0:00      \_ sshd: ubuntu@pts/0
ubuntu    2205  0.0  0.0  25216  5492 pts/0    Ss   11:41   0:00          \_ -bash
root      2320  0.0  0.0  61960  4584 pts/0    S    11:41   0:00              \_ sudo -s
root      2368  0.0  0.0  25272  5464 pts/0    S    11:41   0:00                  \_ /bin/bash
root     12627  0.0  0.0  44400  6296 pts/0    R+   11:53   0:00                      \_ ps fauxww
root      2603  0.0  0.0 117408 24192 ?        Ss   Nov22   0:05 /usr/sbin/apache2 -k start
www-data 26519  0.1  0.1 2432240 46580 ?       Sl   01:00   0:43  \_ /usr/sbin/apache2 -k start
www-data 26520  0.2  0.1 2436640 50844 ?       Sl   01:00   1:26  \_ /usr/sbin/apache2 -k start
www-data 22062  0.1  0.1 2435564 49668 ?       Sl   01:15   1:15  \_ /usr/sbin/apache2 -k start
www-data  7001  0.2  0.1 2436416 50684 ?       Sl   02:02   1:33  \_ /usr/sbin/apache2 -k start
www-data  7051  0.3  0.1 2437848 52092 ?       Sl   02:02   2:00  \_ /usr/sbin/apache2 -k start
www-data  7124  0.3  0.1 2436948 51328 ?       Sl   02:02   1:50  \_ /usr/sbin/apache2 -k start
www-data  7279  0.4  0.1 2439684 53784 ?       Sl   02:02   2:32  \_ /usr/sbin/apache2 -k start
www-data  7350  0.0  0.1 2430252 44244 ?       Sl   02:02   0:22  \_ /usr/sbin/apache2 -k start
www-data  7393  0.1  0.1 2432852 46920 ?       Sl   02:02   0:39  \_ /usr/sbin/apache2 -k start
www-data  2605  0.0  0.0  20280  1760 ?        Ss   Nov22   0:44 /usr/bin/htcacheclean -d 120 -p /var/cache/apache2/mod_cache_disk -l 300M -n
root      2848  0.0  0.0 234844   124 ?        Sl   Nov22   0:00 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid
lxd       3031  0.0  0.0  50244   724 ?        S    Nov22   0:50 dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.pid --except-interface=lo --no-ping --interface=lxdbr0 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.218.194.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.218.194.2,10.218.194.254,1h -s lxd -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd
lxd       3084  0.0  0.0  50096   148 ?        S    Nov22   0:03 dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/lxd/common/lxd/networks/lxdbr1/dnsmasq.pid --except-interface=lo --no-ping --interface=lxdbr1 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.0.1.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr1/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr1/dnsmasq.hosts --dhcp-range 10.0.1.2,10.0.1.254,1h -s lxd -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr1/dnsmasq.raw -u lxd
lxd       3135  0.0  0.0  49984     4 ?        S    Nov22   0:00 dnsmasq --strict-order --bind-interfaces --pid-file=/var/snap/lxd/common/lxd/networks/lxdbr2/dnsmasq.pid --except-interface=lo --no-ping --interface=lxdbr2 --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.0.2.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr2/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr2/dnsmasq.hosts --dhcp-range 10.0.2.2,10.0.2.254,1h -s lxd -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr2/dnsmasq.raw -u lxd
root      3316  0.0  0.0 293972    96 ?        Ss   Nov22   0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers www-**********
lxduser   3333  0.0  0.0   1512     0 ?        Ss   Nov22   0:00  \_ /sbin/init
lxduser   4207  0.0  0.0   1516     4 ?        Ss   Nov22   0:00      \_ /sbin/syslogd -Z
lxduser   4295  0.0  0.0   1520    20 ?        Ss   Nov22   0:00      \_ /usr/sbin/crond -c /etc/crontabs
lxduser   4469  0.0  0.0   1512     4 ?        Ss   Nov22   0:00      \_ udhcpc -b -p /var/run/udhcpc.eth0.pid -i eth0
lxduser   4855  0.0  0.0   7320     4 ?        S    Nov22   0:00      \_ /usr/sbin/sshd
1000101   5178  0.0  0.0  34996   812 ?        S    Nov22   0:03      \_ /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
1000101   5214  0.0  0.0 102020   264 ?        Ss   Nov22   0:00      |   \_ /usr/bin/php-cgi
1000101   7302  0.0  0.0 104256  8640 ?        S    Nov22   0:27      |       \_ /usr/bin/php-cgi
1000101   7303  0.0  0.0 104256  8792 ?        S    Nov22   0:27      |       \_ /usr/bin/php-cgi
1000101   7304  0.0  0.0 104220  8804 ?        S    Nov22   0:30      |       \_ /usr/bin/php-cgi
1000101   7305  0.0  0.0 104284  8864 ?        S    Nov22   0:29      |       \_ /usr/bin/php-cgi
1000100   5775  0.0  0.0 378104  4924 ?        Sl   Nov22   1:05      \_ /usr/bin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock --port=3306
lxduser   5776  0.0  0.0   1512     0 ?        S    Nov22   0:00      \_ logger -t mysqld -p daemon.error
lxduser   8247  0.0  0.0   1512     0 pts/0    Ss+  Nov22   0:00      \_ /sbin/getty 38400 console
root      3424  0.0  0.0 294228   124 ?        Ss   Nov22   0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers www-**********
lxduser   3442  0.0  0.0   1512     0 ?        Ss   Nov22   0:00  \_ /sbin/init
lxduser   5202  0.0  0.0   1516     4 ?        Ss   Nov22   0:00      \_ /sbin/syslogd -Z
lxduser   5386  0.0  0.0   1520    24 ?        Ss   Nov22   0:00      \_ /usr/sbin/crond -c /etc/crontabs
lxduser   5619  0.0  0.0   1512     4 ?        Ss   Nov22   0:00      \_ udhcpc -b -p /var/run/udhcpc.eth0.pid -i eth0
lxduser   6145  0.0  0.0   7320     4 ?        S    Nov22   0:00      \_ /usr/sbin/sshd
1000101   6516  0.0  0.0  35088  1040 ?        S    Nov22   0:03      \_ /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
1000101   6534  0.0  0.0 102020   224 ?        Ss   Nov22   0:00      |   \_ /usr/bin/php-cgi
1000101   8611  0.1  0.0 105008 11036 ?        S    Nov22   1:26      |       \_ /usr/bin/php-cgi
1000101   8612  0.1  0.0 105160 11432 ?        S    Nov22   1:25      |       \_ /usr/bin/php-cgi
1000101   8613  0.1  0.0 105152 12436 ?        S    Nov22   1:21      |       \_ /usr/bin/php-cgi
1000101   8614  0.1  0.0 104984 11128 ?        S    Nov22   1:25      |       \_ /usr/bin/php-cgi
1000100   7086  0.0  0.0 378816  5836 ?        Sl   Nov22   1:12      \_ /usr/bin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock --port=3306
lxduser   7087  0.0  0.0   1512     0 ?        S    Nov22   0:00      \_ logger -t mysqld -p daemon.error
lxduser   8649  0.0  0.0   1512     0 ?        Ss+  Nov22   0:00      \_ /sbin/getty 38400 console
root      3569  0.0  0.0 294228   112 ?        Ss   Nov22   0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers www-**********
lxduser   3599  0.0  0.0   1512     0 ?        Ss   Nov22   0:00  \_ /sbin/init
lxduser   5449  0.0  0.0   1516     0 ?        Ss   Nov22   0:00      \_ /sbin/syslogd -Z
lxduser   5623  0.0  0.0   1520     0 ?        Ss   Nov22   0:00      \_ /usr/sbin/crond -c /etc/crontabs
lxduser   5868  0.0  0.0   1512     4 ?        Ss   Nov22   0:00      \_ udhcpc -b -p /var/run/udhcpc.eth0.pid -i eth0
lxduser   6195  0.0  0.0   7320     4 ?        S    Nov22   0:00      \_ /usr/sbin/sshd
1000101   6455  0.0  0.0  35036   972 ?        S    Nov22   0:03      \_ /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
1000101   6485  0.0  0.0 102020   200 ?        Ss   Nov22   0:00      |   \_ /usr/bin/php-cgi
1000101   8493  0.0  0.0 105440 11888 ?        S    Nov22   0:45      |       \_ /usr/bin/php-cgi
1000101   8494  0.0  0.0 104764  9532 ?        S    Nov22   0:40      |       \_ /usr/bin/php-cgi
1000101   8495  0.0  0.0 105420 11904 ?        S    Nov22   0:39      |       \_ /usr/bin/php-cgi
1000101   8496  0.0  0.0 105472 11880 ?        S    Nov22   0:40      |       \_ /usr/bin/php-cgi
1000100   6979  0.0  0.0 378208  4784 ?        Sl   Nov22   1:06      \_ /usr/bin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock --port=3306
lxduser   6981  0.0  0.0   1512     0 ?        S    Nov22   0:00      \_ logger -t mysqld -p daemon.error
lxduser   8648  0.0  0.0   1512     0 ?        Ss+  Nov22   0:00      \_ /sbin/getty 38400 console
root      2190  0.0  0.0 293972  9180 ?        Ss   Nov22   0:00 [lxc monitor] /var/snap/lxd/common/lxd/containers www-**********
lxduser   2367  0.0  0.0   1512     0 ?        Ss   Nov22   0:00  \_ /sbin/init
lxduser   9268  0.0  0.0   1516    32 ?        Ss   Nov22   0:00      \_ /sbin/syslogd -Z
lxduser   9477  0.0  0.0   1520    28 ?        Ss   Nov22   0:00      \_ /usr/sbin/crond -c /etc/crontabs
lxduser   9714  0.0  0.0   1512     4 ?        Ss   Nov22   0:00      \_ udhcpc -b -p /var/run/udhcpc.eth0.pid -i eth0
lxduser  10385  0.0  0.0   7320     4 ?        S    Nov22   0:00      \_ /usr/sbin/sshd
1000101  10799  0.0  0.0  35032   852 ?        S    Nov22   0:02      \_ /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
1000101  10831  0.0  0.0 102020  2524 ?        Ss   Nov22   0:00      |   \_ /usr/bin/php-cgi
1000101  12503  0.0  0.0 105324 10368 ?        S    Nov22   0:47      |       \_ /usr/bin/php-cgi
1000101  12504  0.0  0.0 104116  9044 ?        S    Nov22   0:48      |       \_ /usr/bin/php-cgi
1000101  12506  0.0  0.0 105488 12084 ?        S    Nov22   0:52      |       \_ /usr/bin/php-cgi
1000101  12507  0.0  0.0 105476 12032 ?        S    Nov22   0:48      |       \_ /usr/bin/php-cgi
1000100  11402  0.0  0.0 377848  9892 ?        Sl   Nov22   0:26      \_ /usr/bin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib/mysql/plugin --user=mysql --pid-file=/run/mysqld/mysqld.pid --socket=/run/mysqld/mysqld.sock --port=3306
lxduser  11403  0.0  0.0   1512     0 ?        S    Nov22   0:00      \_ logger -t mysqld -p daemon.error
lxduser  11730  0.0  0.0   1512     0 ?        Ss+  Nov22   0:00      \_ /sbin/getty 38400 console
root      7386  1.1  0.2  90480 73480 ?        S<Ls 00:00   8:08 /usr/bin/atop -R -w /var/log/atop/atop_20191123 600
ubuntu    2033  0.0  0.0  77632  8128 ?        Ss   11:41   0:00 /lib/systemd/systemd --user
ubuntu    2034  0.0  0.0 258388  3276 ?        S    11:41   0:00  \_ (sd-pam)

lepe1 · November 23, 2019, 3:16am

I executed strace -fF lxd --debug --group lxd, these are the last lines: (let me know if you need the full dump)

[pid 28770] write(2, "\33[36mDBUG\33[0m[11-23|12:13:33] Co"..., 76 DBUG[11-23|12:13:33] Connecting to a local LXD over a Unix socket 
<unfinished ...>
[pid 28769] futex(0x1f46900, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 28770] <... write resumed> )       = 76
[pid 28721] futex(0xc0000aa148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28719] <... madvise resumed> )     = 0
[pid 28770] write(2, "\33[36mDBUG\33[0m[11-23|12:13:33] Se"..., 142DBUG[11-23|12:13:33] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
 <unfinished ...>
[pid 28719] madvise(0xc000496000, 8192, MADV_FREE <unfinished ...>
[pid 28770] <... write resumed> )       = 142
[pid 28719] <... madvise resumed> )     = 0
[pid 28719] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28770] futex(0xc0000aa148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28719] <... madvise resumed> )     = 0
[pid 28770] <... futex resumed> )       = 1
[pid 28721] <... futex resumed> )       = 0
[pid 28770] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 28721] futex(0xc0000aa148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] <... socket resumed> )      = 13
[pid 28719] madvise(0xc000490000, 8192, MADV_FREE <unfinished ...>
[pid 28770] setsockopt(13, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid 28719] <... madvise resumed> )     = 0
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28719] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28770] sched_yield( <unfinished ...>
[pid 28719] <... madvise resumed> )     = 0
[pid 28770] <... sched_yield resumed> ) = 0
[pid 28719] madvise(0xc000488000, 8192, MADV_FREE <unfinished ...>
[pid 28770] futex(0x1f46900, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 28719] <... madvise resumed> )     = 0
[pid 28719] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc000480000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc00047c000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc000468000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc000458000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc00044e000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc000434000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc0003d8000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc0003d4000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc0003ca000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc0003ae000, 24576, MADV_FREE) = 0
[pid 28719] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc00039e000, 16384, MADV_FREE) = 0
[pid 28719] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc00039a000, 8192, MADV_FREE) = 0
[pid 28719] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc000390000, 16384, MADV_FREE) = 0
[pid 28719] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE) = 0
[pid 28719] madvise(0xc00036c000, 8192, MADV_FREE) = 0
[pid 28719] futex(0x1f46900, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28769] <... futex resumed> )       = 0
[pid 28719] <... futex resumed> )       = 1
[pid 28769] futex(0x1f46900, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28719] futex(0xc0000aa148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] <... futex resumed> )       = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28770] futex(0x1f46900, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28721] <... futex resumed> )       = 0
[pid 28770] <... futex resumed> )       = 0
[pid 28769] nanosleep({tv_sec=0, tv_nsec=3000},  <unfinished ...>
[pid 28721] nanosleep({tv_sec=0, tv_nsec=3000},  <unfinished ...>
[pid 28770] connect(13, {sa_family=AF_UNIX, sun_path="/var/snap/lxd/common/lxd/unix.socket"}, 39 <unfinished ...>
[pid 28769] <... nanosleep resumed> NULL) = 0
[pid 28770] <... connect resumed> )     = -1 ECONNREFUSED (Connection refused)
[pid 28769] sched_yield( <unfinished ...>
[pid 28770] close(13 <unfinished ...>
[pid 28769] <... sched_yield resumed> ) = 0
[pid 28770] <... close resumed> )       = 0
[pid 28769] futex(0x1f44900, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
[pid 28770] newfstatat(AT_FDCWD, "/var/snap/lxd/common/lxd/unix.socket",  <unfinished ...>
[pid 28721] <... nanosleep resumed> NULL) = 0
[pid 28770] <... newfstatat resumed> {st_mode=S_IFSOCK|0660, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid 28721] futex(0xc0000aa148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] write(2, "\33[36mDBUG\33[0m[11-23|12:13:33] De"..., 68DBUG[11-23|12:13:33] Detected stale unix socket, deleting 
 <unfinished ...>
[pid 28719] <... futex resumed> )       = 1
[pid 28770] <... write resumed> )       = 68
[pid 28719] futex(0x1f44900, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] unlinkat(AT_FDCWD, "/var/snap/lxd/common/lxd/unix.socket", 0 <unfinished ...>
[pid 28769] <... futex resumed> )       = 0
[pid 28770] <... unlinkat resumed> )    = 0
[pid 28769] futex(0x1f44900, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 28769] <... futex resumed> )       = 0
[pid 28770] <... socket resumed> )      = 13
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=138551} <unfinished ...>
[pid 28770] setsockopt(13, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid 28719] <... futex resumed> )       = 1
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] setsockopt(13, SOL_SOCKET, SO_REUSEADDR, [1], 4 <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] bind(13, {sa_family=AF_UNIX, sun_path="/var/snap/lxd/common/lxd/unix.socket"}, 39 <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28770] <... bind resumed> )        = 0
[pid 28769] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28770] listen(13, 128 <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] <... listen resumed> )      = 0
[pid 28769] madvise(0xc000534000, 32768, MADV_FREE <unfinished ...>
[pid 28770] epoll_ctl(4, EPOLL_CTL_ADD, 13, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3509415688, u64=140186946989832}} <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] <... epoll_ctl resumed> )   = 0
[pid 28769] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28770] getsockname(13,  <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] <... getsockname resumed> {sa_family=AF_UNIX, sun_path="/var/snap/lxd/common/lxd/unix.socket"}, [112->39]) = 0
[pid 28769] madvise(0xc0004ce000, 24576, MADV_FREE <unfinished ...>
[pid 28770] fchmodat(AT_FDCWD, "/var/snap/lxd/common/lxd/unix.socket", 0660 <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] <... fchmodat resumed> )    = 0
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1711224} <unfinished ...>
[pid 28770] mmap(0x7f7fa8000000, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0 <unfinished ...>
[pid 28719] <... futex resumed> )       = 0
[pid 28770] <... mmap resumed> )        = 0x7f7fa8000000
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] mprotect(0x7f7fa8000000, 135168, PROT_READ|PROT_WRITE <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... mprotect resumed> )    = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=10000000},  <unfinished ...>
[pid 28770] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 14
[pid 28770] connect(14, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 28770] close(14)                   = 0
[pid 28770] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 14
[pid 28770] connect(14, {sa_family=AF_UNIX, sun_path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
[pid 28770] close(14)                   = 0
[pid 28770] open("/etc/group", O_RDONLY|O_CLOEXEC) = 14
[pid 28770] lseek(14, 0, SEEK_CUR)      = 0
[pid 28770] fstat(14, {st_mode=S_IFREG|0644, st_size=867, ...}) = 0
[pid 28770] mmap(NULL, 867, PROT_READ, MAP_SHARED, 14, 0) = 0x7f7fd1395000
[pid 28770] lseek(14, 867, SEEK_SET)    = 867
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] madvise(0xc0004cc000, 8192, MADV_FREE) = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=933456} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] madvise(0xc00045a000, 8192, MADV_FREE) = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1425073} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] madvise(0xc000422000, 8192, MADV_FREE) = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1392256} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] madvise(0xc000404000, 8192, MADV_FREE) = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1446655} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] madvise(0xc0003e0000, 8192, MADV_FREE) = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1420875} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc0000aa148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28721] <... futex resumed> )       = 0
[pid 28769] madvise(0xc0003c8000, 8192, MADV_FREE <unfinished ...>
[pid 28721] futex(0xc0000aa148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28719] <... futex resumed> )       = 1
[pid 28769] futex(0xc0000aa148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28719] nanosleep({tv_sec=0, tv_nsec=3000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28721] <... futex resumed> )       = 0
[pid 28769] futex(0x1f44a20, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1192293} <unfinished ...>
[pid 28721] futex(0xc0000aa148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28719] <... nanosleep resumed> NULL) = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] madvise(0xc0003c6000, 8192, MADV_FREE <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1028292} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] madvise(0xc0003c4000, 8192, MADV_FREE <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=969506} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] madvise(0xc0003c2000, 8192, MADV_FREE <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=910091} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] madvise(0xc0003c0000, 8192, MADV_FREE <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=835744} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] madvise(0xc0003be000, 8192, MADV_FREE <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=876197} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] madvise(0xc0003bc000, 8192, MADV_FREE <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=825735} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc0000aa148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28721] <... futex resumed> )       = 0
[pid 28769] madvise(0xc0003ba000, 8192, MADV_FREE <unfinished ...>
[pid 28721] futex(0xc0000aa148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28719] <... futex resumed> )       = 1
[pid 28769] futex(0xc0000aa148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28719] nanosleep({tv_sec=0, tv_nsec=3000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28721] <... futex resumed> )       = 0
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=927772} <unfinished ...>
[pid 28721] futex(0xc0000aa148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28719] <... nanosleep resumed> NULL) = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] madvise(0xc0003b8000, 8192, MADV_FREE <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1054211} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28770] munmap(0x7f7fd1395000, 867) = 0
[pid 28770] close(14)                   = 0
[pid 28770] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28770] getuid( <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] <... getuid resumed> )      = 0
[pid 28770] fchownat(AT_FDCWD, "/var/snap/lxd/common/lxd/unix.socket", 0, 999, 0 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... fchownat resumed> )    = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] newfstatat(AT_FDCWD, "/var/snap/lxd/common/lxd/devlxd/sock", {st_mode=S_IFSOCK|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] write(2, "\33[36mDBUG\33[0m[11-23|12:13:33] De"..., 68DBUG[11-23|12:13:33] Detected stale unix socket, deleting 
 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] <... write resumed> )       = 68
[pid 28770] unlinkat(AT_FDCWD, "/var/snap/lxd/common/lxd/devlxd/sock", 0 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... unlinkat resumed> )    = 0
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] <... socket resumed> )      = 14
[pid 28769] <... futex resumed> )       = 1
[pid 28770] setsockopt(14, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] setsockopt(14, SOL_SOCKET, SO_REUSEADDR, [1], 4 <unfinished ...>
[pid 28769] madvise(0xc0003b6000, 8192, MADV_FREE <unfinished ...>
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] bind(14, {sa_family=AF_UNIX, sun_path="/var/snap/lxd/common/lxd/devlxd/sock"}, 39 <unfinished ...>
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1265809} <unfinished ...>
[pid 28770] <... bind resumed> )        = 0
[pid 28719] <... futex resumed> )       = 0
[pid 28770] listen(14, 128 <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] <... listen resumed> )      = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] epoll_ctl(4, EPOLL_CTL_ADD, 14, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3509415272, u64=140186946989416}} <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... epoll_ctl resumed> )   = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] getsockname(14, {sa_family=AF_UNIX, sun_path="/var/snap/lxd/common/lxd/devlxd/sock"}, [112->39]) = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] fchmodat(AT_FDCWD, "/var/snap/lxd/common/lxd/devlxd/sock", 0666 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] <... fchmodat resumed> )    = 0
[pid 28770] socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_TCP <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... socket resumed> )      = 15
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] close(15)                   = 0
[pid 28770] socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_TCP <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... socket resumed> )      = 15
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] setsockopt(15, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
[pid 28770] bind(15, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... bind resumed> )        = 0
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_TCP <unfinished ...>
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] <... socket resumed> )      = 16
[pid 28769] <... futex resumed> )       = 1
[pid 28770] setsockopt(16, SOL_IPV6, IPV6_V6ONLY, [0], 4 <unfinished ...>
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] bind(16, {sa_family=AF_INET6, sin6_port=htons(0), inet_pton(AF_INET6, "::ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28 <unfinished ...>
[pid 28769] madvise(0xc0003b4000, 8192, MADV_FREE <unfinished ...>
[pid 28770] <... bind resumed> )        = 0
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] close(16 <unfinished ...>
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1161818} <unfinished ...>
[pid 28770] <... close resumed> )       = 0
[pid 28719] <... futex resumed> )       = 0
[pid 28770] close(15 <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] <... close resumed> )       = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, IPPROTO_IP) = 15
[pid 28770] setsockopt(15, SOL_IPV6, IPV6_V6ONLY, [0], 4 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] setsockopt(15, SOL_SOCKET, SO_BROADCAST, [1], 4) = 0
[pid 28770] setsockopt(15, SOL_SOCKET, SO_REUSEADDR, [1], 4 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] bind(15, {sa_family=AF_INET6, sin6_port=htons(8443), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... bind resumed> )        = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] listen(15, 128)             = 0
[pid 28770] epoll_ctl(4, EPOLL_CTL_ADD, 15, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3509415064, u64=140186946989208}} <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28770] <... epoll_ctl resumed> )   = 0
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28770] getsockname(15,  <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] <... getsockname resumed> {sa_family=AF_INET6, sin6_port=htons(8443), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, [112->28]) = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] madvise(0xc000200000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28770] write(2, "\33[32mINFO\33[0m[11-23|12:13:33] St"..., 58 INFO[11-23|12:13:33] Starting /dev/lxd handler: 
<unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] <... write resumed> )       = 58
[pid 28769] madvise(0xc00039c000, 8192, MADV_FREE <unfinished ...>
[pid 28770] write(2, "\33[32mINFO\33[0m[11-23|12:13:33]  -"..., 124 INFO[11-23|12:13:33]  - binding devlxd socket                 socket=/var/snap/lxd/common/lxd/devlxd/sock
<unfinished ...>
[pid 28769] <... madvise resumed> )     = 0
[pid 28770] <... write resumed> )       = 124
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] write(2, "\33[32mINFO\33[0m[11-23|12:13:33] RE"..., 48INFO[11-23|12:13:33] REST API daemon: 
 <unfinished ...>
[pid 28769] <... futex resumed> )       = 0
[pid 28770] <... write resumed> )       = 48
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=943473} <unfinished ...>
[pid 28770] write(2, "\33[32mINFO\33[0m[11-23|12:13:33]  -"..., 124INFO[11-23|12:13:33]  - binding Unix socket                   socket=/var/snap/lxd/common/lxd/unix.socket
 <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] <... write resumed> )       = 124
[pid 28719] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid 28770] write(2, "\33[32mINFO\33[0m[11-23|12:13:33]  -"..., 97INFO[11-23|12:13:33]  - binding TCP socket                    socket=[::]:8443
 <unfinished ...>
[pid 28719] futex(0xc0000aa148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] <... write resumed> )       = 97
[pid 28721] <... futex resumed> )       = 0
[pid 28719] <... futex resumed> )       = 1
[pid 28770] fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1} <unfinished ...>
[pid 28721] futex(0xc0004f4148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] <... fcntl resumed> )       = 0
[pid 28771] <... futex resumed> )       = 0
[pid 28770] fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510} <unfinished ...>
[pid 28771] futex(0xc0000aa848, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] <... fcntl resumed> )       = 0
[pid 28771] <... futex resumed> )       = 1
[pid 28770] fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1} <unfinished ...>
[pid 28771] accept4(15,  <unfinished ...>
[pid 28770] <... fcntl resumed> )       = 0
[pid 28771] <... accept4 resumed> 0xc00011b9f0, [112], SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28770] stat("/var/snap/lxd/common/lxd/database/local.db-journal",  <unfinished ...>
[pid 28771] futex(0xc0004f4148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] <... stat resumed> 0x7f7fba2dc630) = -1 ENOENT (No such file or directory)
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] pread64(3,  <unfinished ...>
[pid 28769] futex(0x1f44aa0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=96029} <unfinished ...>
[pid 28770] <... pread64 resumed> "\0\0\0#\0\0\0\n\0\0\0\0\0\0\0\0", 16, 24) = 16
[pid 28725] <... futex resumed> )       = 0
[pid 28770] stat("/var/snap/lxd/common/lxd/database/local.db-wal",  <unfinished ...>
[pid 28725] futex(0xc0000aa848, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] <... stat resumed> 0x7f7fba2dc630) = -1 ENOENT (No such file or directory)
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] fstat(3,  <unfinished ...>
[pid 28769] futex(0xc0000aa848, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] <... fstat resumed> {st_mode=S_IFREG|0644, st_size=40960, ...}) = 0
[pid 28769] <... futex resumed> )       = 1
[pid 28770] fcntl(3, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=1073741825, l_len=1} <unfinished ...>
[pid 28769] futex(0x1f44b20, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=195895} <unfinished ...>
[pid 28770] <... fcntl resumed> )       = 0
[pid 28725] <... futex resumed> )       = 0
[pid 28770] fcntl(3, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=1} <unfinished ...>
[pid 28725] futex(0xc0000aa848, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] <... fcntl resumed> )       = 0
[pid 28721] <... futex resumed> )       = 1
[pid 28770] fcntl(3, F_SETLK, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] <... fcntl resumed> )       = 0
[pid 28769] futex(0xc0000aa848, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28721] accept4(13,  <unfinished ...>
[pid 28770] pread64(3,  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28770] <... pread64 resumed> "\r\0\0\0\0\20\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096, 32768) = 4096
[pid 28769] futex(0x1f44aa0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=399261} <unfinished ...>
[pid 28770] fcntl(3, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=1073741826, l_len=510} <unfinished ...>
[pid 28725] <... futex resumed> )       = 0
[pid 28770] <... fcntl resumed> )       = 0
[pid 28725] futex(0xc0004f4148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=1073741824, l_len=2} <unfinished ...>
[pid 28771] <... futex resumed> )       = 0
[pid 28770] <... fcntl resumed> )       = 0
[pid 28771] futex(0xc0000aa4c8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_SET, l_start=0, l_len=0} <unfinished ...>
[pid 28771] <... futex resumed> )       = 1
[pid 28770] <... fcntl resumed> )       = 0
[pid 28771] futex(0xc0004f4148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] write(2, "\33[32mINFO\33[0m[11-23|12:13:33] In"..., 60INFO[11-23|12:13:33] Initializing global database 
 <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] <... write resumed> )       = 60
[pid 28769] futex(0x1f44aa0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=799585} <unfinished ...>
[pid 28770] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0 <unfinished ...>
[pid 28725] <... futex resumed> )       = 1
[pid 28770] <... socket resumed> )      = 16
[pid 28725] nanosleep({tv_sec=0, tv_nsec=3000},  <unfinished ...>
[pid 28770] setsockopt(16, SOL_SOCKET, SO_BROADCAST, [1], 4 <unfinished ...>
[pid 28725] <... nanosleep resumed> NULL) = 0
[pid 28770] <... setsockopt resumed> )  = 0
[pid 28725] futex(0x1f44a20, FUTEX_WAIT_PRIVATE, 0, {tv_sec=4, tv_nsec=999721199} <unfinished ...>
[pid 28770] connect(16, {sa_family=AF_UNIX, sun_path=@"00081"}, 8 <unfinished ...>
[pid 28723] <... futex resumed> )       = 0
[pid 28770] <... connect resumed> )     = 0
[pid 28731] <... epoll_wait resumed> [{EPOLLIN, {u32=11, u64=11}}], 1024, 500) = 1
[pid 28770] epoll_ctl(4, EPOLL_CTL_ADD, 16, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=3509415480, u64=140186946989624}} <unfinished ...>
[pid 28731] accept4(11, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK <unfinished ...>
[pid 28559] <... epoll_pwait resumed> [{EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28770] <... epoll_ctl resumed> )   = 0
[pid 28559] epoll_pwait(4,  <unfinished ...>
[pid 28770] getsockname(16,  <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28770] <... getsockname resumed> {sa_family=AF_UNIX}, [112->2]) = 0
[pid 28769] futex(0xc0000aa4c8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28770] getpeername(16,  <unfinished ...>
[pid 28769] <... futex resumed> )       = 0
[pid 28770] <... getpeername resumed> {sa_family=AF_UNIX, sun_path=@"00081"}, [112->8]) = 0
[pid 28769] futex(0x1f44ba0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=1594909} <unfinished ...>
[pid 28770] write(16, "\1\0\0\0\0\0\0\0", 8 <unfinished ...>
[pid 28731] <... accept4 resumed> )     = 17
[pid 28770] <... write resumed> )       = 8
[pid 28731] getsockopt(17, SOL_SOCKET, SO_PEERCRED,  <unfinished ...>
[pid 28770] write(16, "\1\0\0\0\0\0\0\0", 8 <unfinished ...>
[pid 28731] <... getsockopt resumed> {pid=28559, uid=0, gid=0}, [12]) = 0
[pid 28770] <... write resumed> )       = 8
[pid 28731] accept4(11, NULL, NULL, SOCK_CLOEXEC|SOCK_NONBLOCK <unfinished ...>
[pid 28770] write(16, "\0\0\0\0\0\0\0\0", 8 <unfinished ...>
[pid 28731] <... accept4 resumed> )     = -1 EAGAIN (Resource temporarily unavailable)
[pid 28770] <... write resumed> )       = 8
[pid 28723] futex(0xc0000aa4c8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28770] read(16,  <unfinished ...>
[pid 28731] epoll_ctl(7, EPOLL_CTL_ADD, 17, {EPOLLIN, {u32=17, u64=17}} <unfinished ...>
[pid 28770] <... read resumed> 0xc00056ea58, 8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28731] <... epoll_ctl resumed> )   = 0
[pid 28770] futex(0xc0004be148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28731] epoll_wait(7,  <unfinished ...>
[pid 28723] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid 28731] <... epoll_wait resumed> [{EPOLLIN, {u32=17, u64=17}}], 1024, 418) = 1
[pid 28723] futex(0xc0000aa4c8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28731] read(17,  <unfinished ...>
[pid 28721] <... accept4 resumed> 0xc00012ba80, [112], SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28559] <... epoll_pwait resumed> [{EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28731] <... read resumed> "\1\0\0\0\0\0\0\0", 8) = 8
[pid 28559] epoll_pwait(4,  <unfinished ...>
[pid 28731] read(17,  <unfinished ...>
[pid 28721] futex(0xc0000aa148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28559] <... epoll_pwait resumed> [{EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28731] <... read resumed> "\1\0\0\0\0\0\0\0", 8) = 8
[pid 28559] epoll_pwait(4,  <unfinished ...>
[pid 28731] read(17,  <unfinished ...>
[pid 28719] accept4(14,  <unfinished ...>
[pid 28559] <... epoll_pwait resumed> [{EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28559] epoll_pwait(4,  <unfinished ...>
[pid 28769] futex(0xc0000aa148, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28731] <... read resumed> "\0\0\0\0\0\0\0\0", 8) = 8
[pid 28769] <... futex resumed> )       = 1
[pid 28731] write(17, "\2\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0001\0\0\0\0\0\0\0", 24 <unfinished ...>
[pid 28769] futex(0x1f449a0, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=3199435} <unfinished ...>
[pid 28559] <... epoll_pwait resumed> [{EPOLLIN|EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28731] <... write resumed> )       = 24
[pid 28559] read(16,  <unfinished ...>
[pid 28731] epoll_ctl(7, EPOLL_CTL_ADD, 17, {EPOLLIN, {u32=17, u64=17}} <unfinished ...>
[pid 28559] <... read resumed> "\2\0\0\0\1\0\0\0", 8) = 8
[pid 28731] <... epoll_ctl resumed> )   = -1 EEXIST (File exists)
[pid 28559] read(16,  <unfinished ...>
[pid 28731] epoll_ctl(7, EPOLL_CTL_MOD, 17, {EPOLLIN, {u32=17, u64=17}} <unfinished ...>
[pid 28559] <... read resumed> "\1\0\0\0\0\0\0\0001\0\0\0\0\0\0\0", 16) = 16
[pid 28731] <... epoll_ctl resumed> )   = 0
[pid 28559] write(16, "\1\0\0\0\1\0\0\0", 8 <unfinished ...>
[pid 28731] epoll_wait(7,  <unfinished ...>
[pid 28559] <... write resumed> )       = 8
[pid 28731] <... epoll_wait resumed> [{EPOLLIN, {u32=17, u64=17}}], 1024, 417) = 1
[pid 28559] write(16, "\0\0\0\0\0\0\0\0", 8 <unfinished ...>
[pid 28731] read(17,  <unfinished ...>
[pid 28559] <... write resumed> )       = 8
[pid 28731] <... read resumed> "\1\0\0\0\1\0\0\0", 8) = 8
[pid 28559] read(16,  <unfinished ...>
[pid 28731] read(17,  <unfinished ...>
[pid 28559] <... read resumed> 0xc00056ea58, 8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28731] <... read resumed> "\0\0\0\0\0\0\0\0", 8) = 8
[pid 28559] epoll_pwait(4,  <unfinished ...>
[pid 28731] write(17, "\1\0\0\0\2\0\0\0\230:\0\0\0\0\0\0", 16 <unfinished ...>
[pid 28559] <... epoll_pwait resumed> [{EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, 0, NULL, 0) = 1
[pid 28731] <... write resumed> )       = 16
[pid 28559] epoll_pwait(4,  <unfinished ...>
[pid 28731] epoll_ctl(7, EPOLL_CTL_ADD, 17, {EPOLLIN, {u32=17, u64=17}} <unfinished ...>
[pid 28559] <... epoll_pwait resumed> [{EPOLLIN|EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28731] <... epoll_ctl resumed> )   = -1 EEXIST (File exists)
[pid 28559] read(16,  <unfinished ...>
[pid 28731] epoll_ctl(7, EPOLL_CTL_MOD, 17, {EPOLLIN, {u32=17, u64=17}} <unfinished ...>
[pid 28559] <... read resumed> "\1\0\0\0\2\0\0\0", 8) = 8
[pid 28731] <... epoll_ctl resumed> )   = 0
[pid 28559] read(16,  <unfinished ...>
[pid 28731] epoll_wait(7,  <unfinished ...>
[pid 28559] <... read resumed> "\230:\0\0\0\0\0\0", 8) = 8
[pid 28721] <... futex resumed> )       = 0
[pid 28559] write(2, "\33[36mDBUG\33[0m[11-23|12:13:33] Dq"..., 69DBUG[11-23|12:13:33] Dqlite: connected address=1 attempt=0 
 <unfinished ...>
[pid 28721] epoll_pwait(4,  <unfinished ...>
[pid 28559] <... write resumed> )       = 69
[pid 28721] <... epoll_pwait resumed> [], 128, 0, NULL, 0) = 0
[pid 28559] write(16, "\4\0\0\0\3\0\0\0", 8 <unfinished ...>
[pid 28721] epoll_pwait(4,  <unfinished ...>
[pid 28559] <... write resumed> )       = 8
[pid 28731] <... epoll_wait resumed> [{EPOLLIN, {u32=17, u64=17}}], 1024, 416) = 1
[pid 28559] write(16, "db.bin\0\0\0\0\0\0\0\0\0\0volatile\0\0\0\0\0\0\0\0", 32 <unfinished ...>
[pid 28731] read(17,  <unfinished ...>
[pid 28559] <... write resumed> )       = 32
[pid 28731] <... read resumed> "\4\0\0\0\3\0\0\0", 8) = 8
[pid 28559] read(16,  <unfinished ...>
[pid 28731] read(17,  <unfinished ...>
[pid 28559] <... read resumed> 0xc000038230, 8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28731] <... read resumed> "db.bin\0\0\0\0\0\0\0\0\0\0volatile\0\0\0\0\0\0\0\0", 32) = 32
[pid 28559] futex(0x1f41508, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28731] mprotect(0x7f7fb3512000, 1052672, PROT_READ|PROT_WRITE <unfinished ...>
[pid 28721] <... epoll_pwait resumed> [{EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28731] <... mprotect resumed> )    = 0
[pid 28721] epoll_pwait(4,  <unfinished ...>
[pid 28719] <... accept4 resumed> 0xc00011fa80, [112], SOCK_CLOEXEC|SOCK_NONBLOCK) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28721] <... epoll_pwait resumed> [{EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28721] epoll_pwait(4,  <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28769] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 28769] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28717] <... futex resumed> )       = 0
[pid 28769] futex(0xc00007abc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28769] <... futex resumed> )       = 1
[pid 28719] <... futex resumed> )       = 0
[pid 28769] futex(0x1f44920, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=6399349} <unfinished ...>
[pid 28719] futex(0xc00007abc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28731] write(17, "\1\0\0\0\4\0\0\0\0\0\0\0\0\0\0\0", 16) = 16
[pid 28721] <... epoll_pwait resumed> [{EPOLLIN|EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, -1, NULL, 3) = 1
[pid 28731] epoll_ctl(7, EPOLL_CTL_ADD, 17, {EPOLLIN, {u32=17, u64=17}} <unfinished ...>
[pid 28721] futex(0x1f40870, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28731] <... epoll_ctl resumed> )   = -1 EEXIST (File exists)
[pid 28721] <... futex resumed> )       = 1
[pid 28731] epoll_ctl(7, EPOLL_CTL_MOD, 17, {EPOLLIN, {u32=17, u64=17}} <unfinished ...>
[pid 28721] read(16,  <unfinished ...>
[pid 28731] <... epoll_ctl resumed> )   = 0
[pid 28721] <... read resumed> "\1\0\0\0\4\0\0\0", 8) = 8
[pid 28731] epoll_wait(7,  <unfinished ...>
[pid 28721] read(16,  <unfinished ...>
[pid 28717] <... futex resumed> )       = 0
[pid 28721] <... read resumed> "\0\0\0\0\0\0\0\0", 8) = 8
[pid 28717] nanosleep({tv_sec=0, tv_nsec=20000},  <unfinished ...>
[pid 28721] write(16, "\5\0\0\0\10\0\0\0", 8 <unfinished ...>
[pid 28731] <... epoll_wait resumed> [{EPOLLIN, {u32=17, u64=17}}], 1024, 413) = 1
[pid 28721] <... write resumed> )       = 8
[pid 28731] read(17,  <unfinished ...>
[pid 28721] write(16, "\0\0\0\0\0\0\0\0PRAGMA cache_size=-50000"..., 40 <unfinished ...>
[pid 28731] <... read resumed> "\5\0\0\0\10\0\0\0", 8) = 8
[pid 28721] <... write resumed> )       = 40
[pid 28731] read(17,  <unfinished ...>
[pid 28721] read(16,  <unfinished ...>
[pid 28731] <... read resumed> "\0\0\0\0\0\0\0\0PRAGMA cache_size=-50000"..., 40) = 40
[pid 28721] <... read resumed> 0xc000038230, 8) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28731] eventfd2(0, EFD_NONBLOCK <unfinished ...>
[pid 28721] epoll_pwait(4,  <unfinished ...>
[pid 28731] <... eventfd2 resumed> )    = 18
[pid 28721] <... epoll_pwait resumed> [{EPOLLOUT, {u32=3509415480, u64=140186946989624}}], 128, 0, NULL, 0) = 1
[pid 28731] ioctl(18, FIONBIO, [1] <unfinished ...>
[pid 28721] epoll_pwait(4,  <unfinished ...>
[pid 28731] <... ioctl resumed> )       = 0
[pid 28717] <... nanosleep resumed> NULL) = 0
[pid 28731] open("/var/snap/lxd/common/lxd/database/global/open-1", O_WRONLY|O_CREAT|O_EXCL|O_DSYNC, 0600 <unfinished ...>
[pid 28717] futex(0x1f40870, FUTEX_WAIT_PRIVATE, 0, {tv_sec=60, tv_nsec=0} <unfinished ...>
[pid 28731] <... open resumed> )        = 19
[pid 28731] io_setup(1, 0x7f7fb0f092a0) = -1 EAGAIN (Resource temporarily unavailable)
[pid 28731] close(19)                   = 0
[pid 28731] unlink("/var/snap/lxd/common/lxd/database/global/open-1") = 0
[pid 28731] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x940} ---
[pid 28731] rt_sigprocmask(SIG_SETMASK, NULL, ~[KILL STOP], 8) = 0
[pid 28731] rt_sigprocmask(SIG_SETMASK, ~[], NULL, 8) = 0
[pid 28731] sigaltstack(NULL, {ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=0}) = 0
[pid 28731] sigaltstack({ss_sp=0xc00007c000, ss_flags=0, ss_size=32768}, NULL) = 0
[pid 28731] rt_sigprocmask(SIG_SETMASK, ~[HUP INT QUIT ILL TRAP ABRT BUS FPE KILL SEGV TERM STKFLT CHLD STOP PROF SYS RTMIN RT_1], NULL, 8) = 0
[pid 28731] gettid()                    = 28731
[pid 28731] rt_sigprocmask(SIG_UNBLOCK, [SEGV], NULL, 8) = 0
[pid 28731] rt_sigaction(SIGSEGV, {sa_handler=SIG_DFL, sa_mask=~[], sa_flags=SA_RESTORER|SA_ONSTACK|SA_RESTART|SA_SIGINFO, sa_restorer=0x7f7fd0874390}, NULL, 8) = 0
[pid 28731] getpid()                    = 28559
[pid 28731] gettid()                    = 28731
[pid 28731] tgkill(28559, 28731, SIGSEGV) = 0
[pid 28731] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_TKILL, si_pid=28559, si_uid=0} ---
[pid 28559] <... futex resumed>)        = ?
[pid 28771] <... futex resumed>)        = ?
[pid 28770] <... futex resumed>)        = ?
[pid 28769] <... futex resumed>)        = ?
[pid 28725] <... futex resumed>)        = ?
[pid 28724] <... futex resumed>)        = ?
[pid 28723] <... futex resumed>)        = ?
[pid 28722] <... futex resumed>)        = ?
[pid 28721] <... epoll_pwait resumed> <unfinished ...>) = ?
[pid 28720] <... futex resumed>)        = ?
[pid 28719] <... futex resumed>)        = ?
[pid 28718] <... futex resumed>)        = ?
[pid 28717] <... futex resumed>)        = ?
[pid 28720] +++ killed by SIGSEGV +++
[pid 28771] +++ killed by SIGSEGV +++
[pid 28770] +++ killed by SIGSEGV +++
[pid 28769] +++ killed by SIGSEGV +++
[pid 28725] +++ killed by SIGSEGV +++
[pid 28724] +++ killed by SIGSEGV +++
[pid 28723] +++ killed by SIGSEGV +++
[pid 28722] +++ killed by SIGSEGV +++
[pid 28721] +++ killed by SIGSEGV +++
[pid 28719] +++ killed by SIGSEGV +++
[pid 28718] +++ killed by SIGSEGV +++
[pid 28717] +++ killed by SIGSEGV +++
[pid 28731] +++ killed by SIGSEGV +++
+++ killed by SIGSEGV +++
Segmentation fault

From what I can see, thre is one: ECONNREFUSED (Connection refused) to the socket (/var/snap/lxd/common/lxd/unix.socket) then followed by several attempts: ETIMEDOUT (Connection timed out), but not sure what is causing the Segmentation fault.

I tried removing the socket, but same result. The command: systemctl start snap.lxd.daemon.unix.socket creates the socket but systemctl status snap.lxd.daemon.unix.socket still returns failed:

● snap.lxd.daemon.unix.socket - Socket unix for snap application lxd.daemon
   Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.unix.socket; enabled; vendor preset: enabled)
   Active: failed (Result: service-start-limit-hit) since Sat 2019-11-23 12:12:06 JST; 12min ago
   Listen: /var/snap/lxd/common/lxd/unix.socket (Stream)

Nov 23 12:11:39 shiny.webporte.jp systemd[1]: Listening on Socket unix for snap application lxd.daemon.
Nov 23 12:12:06 shiny.webporte.jp systemd[1]: snap.lxd.daemon.unix.socket: Failed with result 'service-start-limit-hit'.

lepe1 · November 23, 2019, 3:45am

Not sure if related, but I if I list /var/snap/lxd/common it shows:

-rw-r--r--  1 root root  190 Nov 22 17:20 config
drwxr-xr-x  2 root root 4.0K Sep 28  2018 lxc/
-rw-------  1 root root    5 Nov 22 15:41 lxcfs.pid
drwx--x--x 15 root root 4.0K Nov 23 12:13 lxd/
-rw-r--r--  1 root root    6 Nov 23 12:12 lxd.pid
lrwxrwxrwx  1 root root   16 Nov 23 12:12 mntns -> /proc/27486/root
drwx------  2 root root   80 Nov 22 15:41 ns/
drwx--x--x  2 root root 4.0K Sep 28  2018 shmounts/
-rw-r--r--  1 root root    8 Nov 23 12:12 state
drwxr-xr-x  3 root root 4.0K Sep 28  2018 var/

The mntns link is broken and the process 27486 doesn’t exists. Also, /var/snap/lxd/common/shmounts/ is empty which makes some other links to be broken, like: /var/snap/lxd/common/shmounts/containers and /var/snap/lxd/common/shmounts/lxcfs.

stgraber · November 23, 2019, 5:10pm

Can you send a tarball of /var/snap/lxd/common/lxd/database to stgraber at ubuntu dot com?

Based on your strace and log output, it looks like it’s the database crashing on startup.
It may be easily recoverable but it’s a failure we’ve not seen before so want to take a look at what happened.

lepe1 · November 24, 2019, 7:46am

I sent you a PM and to your email.

stgraber · November 26, 2019, 3:40am

@freeekanayaka do you see anything odd in that strace which would explain the segfault?
Loading the same database here doesn’t produce it so I’m suspecting something odd going on with the server that’s running that LXD daemon.

freeekanayaka · November 26, 2019, 8:27am

It seems the problem is:

[pid 28731] io_setup(1, 0x7f7fb0f092a0) = -1 EAGAIN (Resource temporarily unavailable)

from the man page of io_setup:

       EAGAIN The specified nr_events exceeds the user's limit of available events, as defined in /proc/sys/fs/aio-max-nr.

@lepe1 what does cat /proc/sys/fs/aio-max-nr show?

It looks pretty odd that you ran out of AIO context slots in the kernel, unless the default value (64k) was changed or you have some workload that consumes a lot of them.

lepe1 · November 26, 2019, 8:49am

Thanks for helping. This is what /proc/sys/fs/aio-max-nr reports: 65536

freeekanayaka · November 26, 2019, 8:53am

If you still have the full strace (or can generate a new one), can you check how many times io_setup is invoked during the process run? It’s extremely unlikely, but it might be that the process itself is eating up all available AIO slots due to some bug.

Otherwise, it might be a kernel bug? That’d be quite odd too, since we never saw this, and I believe this part of kernel code is relatively stable.

Also, can you confirm that this crash happens consistently all the time you try to start the lxd daemon? And that io_setup fails every time?

lepe1 · November 26, 2019, 8:53am

I followed this recommendation: Setting the sysctl parameter aio-max-nr value to 1048576 :

Setting the sysctl parameter aio-max-nr value to 1048576

The Linux kernel provides the Asynchronous non-blocking I/O (AIO) feature that allows a process to initiate multiple I/O operations simultaneously without having to wait for any of them to complete. This helps boost performance for applications that are able to overlap processing and I/O.

The performance can be tuned using the /proc/sys/fs/aio-max-nr virtual file in the proc file system. The aio-max-nr parameter determines the maximum number of allowable concurrent requests.

Another parameter, /proc/sys/fs/aio-nr , provides the current system-wide number of asynchronous requests.

Veritas recommends that you set the aio-max-nr value to 1048576. This helps HyperScale to perform optimally, in an environment that involves heavy I/O workloads.

Perform the following steps on all HyperScale compute and data nodes:

To set the aio-max-nr value, add the following line to the /etc/sysctl.conf file: fs.aio-max-nr = 1048576

To activate the new setting, run the following command: # sysctl -p /etc/sysctl.conf

And I’m able to do lxc list and everything seems fine now.
Thank you so much for you help!

freeekanayaka · November 26, 2019, 8:56am

Nice! However I’m quite puzzled about why you needed to bump that value. What else is consuming AIO request slots on your system? So much to run out of the default 64k slots.

lepe1 · November 26, 2019, 8:57am

If you want me to send you more information to know the reason, I’m more than happy to assist. Perhaps could it be that this server is using a lot of swap space? (due to low memory at the moment).

I will conduct my own research about the topic and post here if I find something.

freeekanayaka · November 26, 2019, 9:01am

Hm, I don’t think swap should matter. But wait, I asked you to show the wrong value, actually could you paste the output of:

cat /proc/sys/fs/aio-nr

Also, could you try to set the value of fs.aio-max-nr back to the original 65536 and see if you can reproduce the issue (io_setup returning EGAIN) all the times you try to start LXD?

lepe1 · November 26, 2019, 9:03am

The value reported by cat /proc/sys/fs/aio-nr is: 70859
Is there a way to know why that number is so high?

I can confirm that setting it back to 65536 creates the “connection refused” error in LXD.

freeekanayaka · November 26, 2019, 9:44am

@lepe1, bingo. Does the value of aio-nr stay roughly the same if with and without the LXD process running? If it does, it means there is another process consuming them. I’m not aware of any easy way to figure out which process would that be, @brauner or @stgraber might have tricks for that tho… The only way that comes to my mind is to terminate your running processes one by one and see when that number drops back to zero.

lepe1 · November 26, 2019, 9:47am

My guess is that it can be all the processes running inside the 300+ containers. Unfortunately I can’t terminate all processes as its a production server. But the next time we have to shut it down for maintenance (like adding more memory), I will try that. For the meantime, I’m glad it was fixed. Thanks guys!

freeekanayaka · November 26, 2019, 10:00am

Just as data point, among those 300+ containers do you have a lot of database processes? E.g. MySQL PostgreSQL? Those might be somehow hungry consumers of the AIO kernel sub-system.

stgraber · November 26, 2019, 2:57pm

@freeekanayaka
If there’s no obvious downsides to bumping aio.max-nr, can you send a PR to update production-setup.md to add that sysctl to the list with a suggested value and description?