Lxc hangs on list and everything

LXD
1

Hi,
I have encountered problem recently - lxc stopped working.
“lxc list” hangs as well as pretty much every other command.

lxc --debug --verbose list
DBUG[09-18|21:54:36] Connecting to a local LXD over a Unix socket 
DBUG[09-18|21:54:36] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=

I see no other errors or warnings that could give me a clue what’s going on

sudo systemctl status snap.lxd.daemon
● snap.lxd.daemon.service - Service for snap application lxd.daemon
   Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static; vendor preset: enabled)
   Active: active (running) since Wed 2019-09-18 21:45:07 CEST; 10min ago
 Main PID: 17186 (daemon.start)
    Tasks: 0 (limit: 4915)
   Memory: 4.8M
   CGroup: /system.slice/snap.lxd.daemon.service
           ‣ 17186 /bin/sh /snap/lxd/11964/commands/daemon.start

wrz 18 21:45:07 alexeys-desktop lxd.daemon[1851]:   8: fd:  16: perf_event
wrz 18 21:45:07 alexeys-desktop lxd.daemon[1851]:   9: fd:  17: cpu,cpuacct
wrz 18 21:45:07 alexeys-desktop lxd.daemon[1851]:  10: fd:  18: memory
wrz 18 21:45:07 alexeys-desktop lxd.daemon[1851]:  11: fd:  19: name=systemd
wrz 18 21:45:07 alexeys-desktop lxd.daemon[1851]:  12: fd:  20: unified
wrz 18 21:45:07 alexeys-desktop lxd.daemon[1851]: lxcfs.c: 152: do_reload: lxcfs: reloaded
wrz 18 21:45:07 alexeys-desktop lxd.daemon[17186]: => Re-using existing LXCFS
wrz 18 21:45:07 alexeys-desktop lxd.daemon[17186]: => Starting LXD
wrz 18 21:45:07 alexeys-desktop lxd.daemon[17186]: t=2019-09-18T21:45:07+0200 lvl=warn msg="Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored."
wrz 18 21:45:07 alexeys-desktop lxd.daemon[17186]: t=2019-09-18T21:45:07+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
2

Hi,

please start your lxd.daemon with the debug flag and try again. Most probably there will be some more informations to give a clue whats going on.

3

/var/snap/lxd/common/lxd/logs/lxd.log may also contain some useful data.

4

Do you know how to do this? I can’t event stop the daemon:

systemctl stop snap.lxd.daemon.service
Warning: Stopping snap.lxd.daemon.service, but it can still be activated by:
  snap.lxd.daemon.unix.socket

But it seems still be running immediately after this:

systemctl status snap.lxd.daemon.service
● snap.lxd.daemon.service - Service for snap application lxd.daemon
   Loaded: loaded (/etc/systemd/system/snap.lxd.daemon.service; static; vendor preset: enabled)
-->Active: active (running) since Thu 2019-09-19 19:22:31 CEST; 4s ago**
 Main PID: 21343 (daemon.start)
    Tasks: 0 (limit: 4915)
   Memory: 4.7M
   CGroup: /system.slice/snap.lxd.daemon.service
           ‣ 21343 /bin/sh /snap/lxd/11964/commands/daemon.start
5

lxd.log has only this:

t=2019-09-19T19:22:31+0200 lvl=info msg="LXD 3.17 is starting in normal mode" path=/var/snap/lxd/common/lxd
t=2019-09-19T19:22:31+0200 lvl=info msg="Kernel uid/gid map:" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - u 0 0 4294967295" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - g 0 0 4294967295" 
t=2019-09-19T19:22:31+0200 lvl=info msg="Configured LXD uid/gid map:" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - u 0 1000000 1000000000" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - g 0 1000000 1000000000" 
t=2019-09-19T19:22:31+0200 lvl=warn msg="Couldn't find the CGroup blkio.weight, I/O weight limits will be ignored." 
t=2019-09-19T19:22:31+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored." 
t=2019-09-19T19:22:31+0200 lvl=info msg="Kernel features:" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - netnsid-based network retrieval: yes" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - uevent injection: yes" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - seccomp listener: yes" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - unprivileged file capabilities: yes" 
t=2019-09-19T19:22:31+0200 lvl=info msg=" - shiftfs support: no" 
t=2019-09-19T19:22:31+0200 lvl=info msg="Initializing local database"
6

Hi,

please allow me to give you a general advice:
you will have to get a bit more familar with general system administration, otherwise the whole virtualization topic will drive you in a deep cave of tears.

To your question:

When i enter the keywords: “activate lxd debug logging” in my favourite search engine, i will get for example:

https://stgraber.org/2017/02/27/lxd-2-0-debugging-and-contributing-to-lxd-1212/

As you will read there at basically very first:

/var/log/lxd/lxd.log

This is the main LXD log file. To avoid filling up your disk very quickly, only log messages marked as INFO, WARNING or ERROR are recorded there by default. You can change that behavior by passing “–debug” to the LXD daemon.

And as you can see in your systemctl status snap.lxd.daemon, the loaded systemd service file is:

/etc/systemd/system/snap.lxd.daemon.service

If you check this file, you will see how the daemon is started. In combination with the information of the link, you will be able to start the daemon in the debug mode.

Also, since you use snap, you could also run

snap info $package like: snap info lxd

There you will also see some informations about how to enable the debugging.

Last but not least, dont forget, what ever your systemd does, you can do it with your shell too.

Just test around a bit, read a bit and you will be successful with your mission.

7

Well, that was helpful in its own way: I’ve found my ‘snap’ isn’t working properly, it is in state of ‘auto-refresh’ for lxd and few other packages and never finishes this job… So fixing snap probably may help with lxd.

8

Hi,

thats great, so as soon as you fixed it and still have problems, just paste the debug logs to give a chance to help you.

And no worry, you are not alone, i have also all kind of trouble with this all stuff. Good luck !

9

Purgin “snap” and reinstalling fixed the problem. All virtual images were lost but not yet big loss. I’m closing the topic.

10

Unfortunately, that’s an extremely common issue.

11

I had such a case that LXD would get stuck and lxc list would get stuck. In my case it was probably some corruption in the database. I did not investigate further and I just did the following, which was reinstalling from scratch but keeping the containers.