Mount fails when starting container. ZFS filesystem already mounted [LXD 3.22]

The fix for the double free thing has been pushed quite a few hours ago.
If you’re on 13901 or higher, then you have the fix.

Obviously not helping with broken systems as those still need to restart the containers or unmount lxcfs from them.

https://github.com/lxc/lxd-pkg-snap/commits/latest-candidate shows the upcoming changes to the snap. Note that something being in there doesn’t mean it’s in stable yet, but it does mean that the next stable update will include it.

I’ve just finished off restarting my LXD hosts and will monitor the situation.

One thing to note is that my SWAP numbers are completely off inside my containers:

On host:

# free -m
              total        used        free      shared  buff/cache   available
Mem:          32167        8416       14805          65        8946       23356
Swap:          8191          35        8156

Inside containers:

# lxc exec container -- free -m
              total        used        free      shared  buff/cache   available
Mem:            953         368         108           0         477         585
Swap:   8796093021254           0 8796093021254

Also, unrelated to LXD, snap changes gives me an error whereas just yesterday it showed me some changes:

# snap changes
error: no changes found

# snap list
Name  Version    Rev    Tracking  Publisher   Notes
core  16-2.43.3  8689   stable    canonical✓  core
go    1.13.8     5364   stable    mwhudson    classic
lxd   3.22       13901  stable    canonical✓  -

Is there anything special I need to do on my end to resolve this SWAP issue? Prior to recent updates, this issue appeared to be present on 18.04 only. But it is now plaguing both platforms.

@brauner can you look into what’s going on with swap?

Not sure what this would be about but I’ve improved the meminfo codepath now that should handle corner-cases better.

Heads up:

After just 6 days of uptime, the /proc mount issue has returned for containers on my 16.04 host. No issues identified yet for containers on my 18.04 host.

grep lxcfs /var/log/syslog please

Apologies for the omission.

$ sudo grep lxcfs /var/log/syslog
Mar 24 07:23:41 lxcserver lxd.daemon[2753]: Closed liblxcfs.so
Mar 24 07:23:41 lxcserver lxd.daemon[2753]: Running destructor lxcfs_exit
Mar 24 07:23:41 lxcserver lxd.daemon[2753]: Running constructor lxcfs_init to reload liblxcfs
Mar 26 06:49:04 lxcserver kernel: [519660.024260] lxcfs[32252]: segfault at 0 ip 00007f02baec6a32 sp 00007f02a1ffa6b0 error 6 in libc-2.23.so[7f02bae50000+1c0000]
Mar 26 17:38:44 lxcserver lxd.daemon[24169]: Running constructor lxcfs_init to reload liblxcfs

And snap version:

$ snap list lxd
Name  Version  Rev    Tracking  Publisher   Notes
lxd   3.23     13987  stable    canonical✓  -

Snap changes:

$ snap changes lxd
ID   Status  Spawn                    Ready                    Summary
233  Done    yesterday at 17:38 AEDT  yesterday at 17:38 AEDT  Auto-refresh snaps "core18", "lxd"

Ok, could you pastebin the entire crash entry that showed up around 06:49 in your syslog?

Interestingly, I see no other entry in the logs related to this crash - just that one line. Other than syslog, is there another log file that may have this info?

Log file kern.log also has just this one line.

Maybe journalctl -u snap.lxd.daemon -n 300 will show more?

Hardly anything to do with the crash in there:

http://paste.ubuntu.com/p/NY6GpWRPdF/

@brauner ideas?

The one line segfault report from the kernel isn’t exactly much to go on…