Hi, I am having issues with the LXD daemon refusing to start.
lxc info shows the following
$ lxc info
Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd-user/unix.socket: connect: connection refused
When I try to restart the service and check journalctl -xe, I get the following
Jan 09 14:56:05 gold systemd[1]: Started Service for snap application lxd.daemon.
░░ Subject: A start job for unit snap.lxd.daemon.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit snap.lxd.daemon.service has finished successfully.
░░
░░ The job identifier is 6922.
Jan 09 14:56:05 gold lxd.daemon[1740567]: => Preparing the system (24175)
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Loading snap configuration
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Setting up mntns symlink (mnt:[4026533551])
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Setting up kmod wrapper
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Preparing /boot
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Preparing a clean copy of /run
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Preparing /run/bin
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Preparing a clean copy of /etc
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Preparing a clean copy of /usr/share/misc
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Setting up ceph configuration
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Setting up LVM configuration
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Setting up OVN configuration
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Rotating logs
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Setting up ZFS (2.1)
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Escaping the systemd cgroups
Jan 09 14:56:05 gold lxd.daemon[1740567]: ====> Detected cgroup V2
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Escaping the systemd process resource limits
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Disabling shiftfs at user request
Jan 09 14:56:05 gold lxd.daemon[25308]: Closed liblxcfs.so
Jan 09 14:56:05 gold lxd.daemon[25308]: Running destructor lxcfs_exit
Jan 09 14:56:05 gold lxd.daemon[25308]: Running constructor lxcfs_init to reload liblxcfs
Jan 09 14:56:05 gold lxd.daemon[25308]: mount namespace: 6
Jan 09 14:56:05 gold lxd.daemon[25308]: hierarchies:
Jan 09 14:56:05 gold lxd.daemon[25308]: 0: fd: 8: cpuset,cpu,io,memory,hugetlb,pids,rdma,misc
Jan 09 14:56:05 gold lxd.daemon[25308]: Kernel supports pidfds
Jan 09 14:56:05 gold lxd.daemon[25308]: Kernel does not support swap accounting
Jan 09 14:56:05 gold lxd.daemon[25308]: api_extensions:
Jan 09 14:56:05 gold lxd.daemon[25308]: - cgroups
Jan 09 14:56:05 gold lxd.daemon[25308]: - sys_cpu_online
Jan 09 14:56:05 gold lxd.daemon[25308]: - proc_cpuinfo
Jan 09 14:56:05 gold lxd.daemon[25308]: - proc_diskstats
Jan 09 14:56:05 gold lxd.daemon[25308]: - proc_loadavg
Jan 09 14:56:05 gold lxd.daemon[25308]: - proc_meminfo
Jan 09 14:56:05 gold lxd.daemon[25308]: - proc_stat
Jan 09 14:56:05 gold lxd.daemon[25308]: - proc_swaps
Jan 09 14:56:05 gold lxd.daemon[25308]: - proc_uptime
Jan 09 14:56:05 gold lxd.daemon[25308]: - proc_slabinfo
Jan 09 14:56:05 gold lxd.daemon[25308]: - shared_pidns
Jan 09 14:56:05 gold lxd.daemon[25308]: - cpuview_daemon
Jan 09 14:56:05 gold lxd.daemon[25308]: - loadavg_daemon
Jan 09 14:56:05 gold lxd.daemon[25308]: - pidfds
Jan 09 14:56:05 gold lxd.daemon[25308]: Reloaded LXCFS
Jan 09 14:56:05 gold lxd.daemon[1740567]: => Re-using existing LXCFS
Jan 09 14:56:05 gold lxd.daemon[1740567]: ==> Reloading LXCFS
Jan 09 14:56:05 gold lxd.daemon[1740567]: => Starting LXD
Jan 09 14:56:06 gold lxd.daemon[1741161]: time="2023-01-09T14:56:06+08:00" level=warning msg=" - Couldn't find the CGroup network priority controller, network priority will be ignored"
Jan 09 14:56:06 gold lxd.daemon[1741161]: time="2023-01-09T14:56:06+08:00" level=warning msg="Instance type not operational" driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
Jan 09 14:56:06 gold lxd.daemon[1741161]: time="2023-01-09T14:56:06+08:00" level=error msg="Failed to start the daemon" err="Bind network address: listen tcp :8443: bind: address already in use"
Jan 09 14:56:06 gold lxd.daemon[1741161]: Error: Bind network address: listen tcp :8443: bind: address already in use
However, sudo netstat -plnta | grep 8443 gives nothing.
It might also be worth noting that I think these issues started after I tried installing juju and bootstrapping with localhost. However, I had issues with that, and ended up snap remove --purge juju. This did not resolve the issue with lxd though, even after a system restart.
I have checked 8443 with lsof -i :8443, there is nothing.
Same goes with sudo netstat -plnta | grep 8443
I have also noticed that when I try to snap remove lxd, followed by snap install lxd, lxc info works again. However, when I do snap restore to restore the data, the problem happens again.
Search these forums for “/var/snap/lxd/common/lxd-user/unix.socket” shows some similar issues for users with custom user ID mapping that isn’t accessible from the snap package.
I have tried to purge lxd and reinstall again. After reinstallation, I can call lxc info fine and it seems to work, but when I try to sudo snap restore to restore the data from an older snapshot, the problem reoccurs.
When I call sudo lxd --debug --group lxd, I get the following output:
sudo lxd --debug --group lxd
INFO [2023-01-10T09:54:22+08:00] LXD is starting mode=normal path=/var/snap/lxd/common/lxd version=5.9
INFO [2023-01-10T09:54:22+08:00] Kernel uid/gid map:
INFO [2023-01-10T09:54:22+08:00] - u 0 0 4294967295
INFO [2023-01-10T09:54:22+08:00] - g 0 0 4294967295
INFO [2023-01-10T09:54:22+08:00] Configured LXD uid/gid map:
INFO [2023-01-10T09:54:22+08:00] - u 0 1000000 1000000000
INFO [2023-01-10T09:54:22+08:00] - g 0 1000000 1000000000
INFO [2023-01-10T09:54:22+08:00] Kernel features:
INFO [2023-01-10T09:54:22+08:00] - closing multiple file descriptors efficiently: yes
INFO [2023-01-10T09:54:22+08:00] - netnsid-based network retrieval: yes
INFO [2023-01-10T09:54:22+08:00] - pidfds: yes
INFO [2023-01-10T09:54:22+08:00] - core scheduling: yes
INFO [2023-01-10T09:54:22+08:00] - uevent injection: yes
INFO [2023-01-10T09:54:22+08:00] - seccomp listener: yes
INFO [2023-01-10T09:54:22+08:00] - seccomp listener continue syscalls: yes
INFO [2023-01-10T09:54:22+08:00] - seccomp listener add file descriptors: yes
INFO [2023-01-10T09:54:22+08:00] - attach to namespaces via pidfds: yes
INFO [2023-01-10T09:54:22+08:00] - safe native terminal allocation : yes
INFO [2023-01-10T09:54:22+08:00] - unprivileged file capabilities: yes
INFO [2023-01-10T09:54:22+08:00] - cgroup layout: cgroup2
WARNING[2023-01-10T09:54:22+08:00] - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored
WARNING[2023-01-10T09:54:22+08:00] - Couldn't find the CGroup network priority controller, network priority will be ignored
INFO [2023-01-10T09:54:22+08:00] - shiftfs support: yes
INFO [2023-01-10T09:54:22+08:00] - idmapped mounts kernel support: yes
WARNING[2023-01-10T09:54:22+08:00] Instance type not operational driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
INFO [2023-01-10T09:54:22+08:00] Instance type operational driver=lxc features="[]" type=container
INFO [2023-01-10T09:54:22+08:00] Initializing local database
DEBUG [2023-01-10T09:54:22+08:00] Refreshing local trusted certificate cache
INFO [2023-01-10T09:54:22+08:00] Set client certificate to server certificate fingerprint=0873aa75a432f295b901487cad44f681ffc1ac9f5e059109efd73399eef4c9b6
DEBUG [2023-01-10T09:54:22+08:00] Initializing database gateway
INFO [2023-01-10T09:54:22+08:00] Starting database node id=1 local=1 role=voter
WARNING[2023-01-10T09:54:23+08:00] Failed setting up shared mounts err="no such file or directory"
INFO [2023-01-10T09:54:23+08:00] Loading daemon configuration
INFO [2023-01-10T09:54:23+08:00] Closing socket socket="[::]:8443" type="REST API TCP socket"
INFO [2023-01-10T09:54:23+08:00] Closing socket socket=/var/snap/lxd/common/lxd/unix.socket type="REST API Unix socket"
INFO [2023-01-10T09:54:23+08:00] Closing socket socket=/var/snap/lxd/common/lxd/devlxd/sock type="devlxd socket"
ERROR [2023-01-10T09:54:23+08:00] Failed to start the daemon err="Bind network address: listen tcp :8443: bind: address already in use"
INFO [2023-01-10T09:54:23+08:00] Starting shutdown sequence signal=interrupt
DEBUG [2023-01-10T09:54:23+08:00] Cancel ongoing or future gRPC connection attempts
INFO [2023-01-10T09:54:23+08:00] Stop database gateway
INFO [2023-01-10T09:54:23+08:00] Not unmounting temporary filesystems (instances are still running)
INFO [2023-01-10T09:54:23+08:00] Daemon stopped
Error: Bind network address: listen tcp :8443: bind: address already in use
Is there any way that I can save my profiles, containers, storage volumes, and config, and possibly import these into a fresh lxd installation without using snap restore?
Alternatively, for a snap lxd installation, is there any way that I might be able to modify lxd conf files to get it to try to bind to an alternative port, and see if that might fix the problem?
Running lxc as non-root has generally not been an issue for me, however when I run lxc as root, I see the following issue:
sudo lxc config show
Error: Get "http://unix.socket/1.0": dial unix /var/snap/lxd/common/lxd/unix.socket: connect: no such file or directory
I would suppose that happens because the daemon isn’t running:
sudo lxd --debug --group lxd
INFO [2023-01-10T09:54:22+08:00] LXD is starting mode=normal path=/var/snap/lxd/common/lxd version=5.9
INFO [2023-01-10T09:54:22+08:00] Kernel uid/gid map:
INFO [2023-01-10T09:54:22+08:00] - u 0 0 4294967295
INFO [2023-01-10T09:54:22+08:00] - g 0 0 4294967295
INFO [2023-01-10T09:54:22+08:00] Configured LXD uid/gid map:
INFO [2023-01-10T09:54:22+08:00] - u 0 1000000 1000000000
INFO [2023-01-10T09:54:22+08:00] - g 0 1000000 1000000000
INFO [2023-01-10T09:54:22+08:00] Kernel features:
INFO [2023-01-10T09:54:22+08:00] - closing multiple file descriptors efficiently: yes
INFO [2023-01-10T09:54:22+08:00] - netnsid-based network retrieval: yes
INFO [2023-01-10T09:54:22+08:00] - pidfds: yes
INFO [2023-01-10T09:54:22+08:00] - core scheduling: yes
INFO [2023-01-10T09:54:22+08:00] - uevent injection: yes
INFO [2023-01-10T09:54:22+08:00] - seccomp listener: yes
INFO [2023-01-10T09:54:22+08:00] - seccomp listener continue syscalls: yes
INFO [2023-01-10T09:54:22+08:00] - seccomp listener add file descriptors: yes
INFO [2023-01-10T09:54:22+08:00] - attach to namespaces via pidfds: yes
INFO [2023-01-10T09:54:22+08:00] - safe native terminal allocation : yes
INFO [2023-01-10T09:54:22+08:00] - unprivileged file capabilities: yes
INFO [2023-01-10T09:54:22+08:00] - cgroup layout: cgroup2
WARNING[2023-01-10T09:54:22+08:00] - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored
WARNING[2023-01-10T09:54:22+08:00] - Couldn't find the CGroup network priority controller, network priority will be ignored
INFO [2023-01-10T09:54:22+08:00] - shiftfs support: yes
INFO [2023-01-10T09:54:22+08:00] - idmapped mounts kernel support: yes
WARNING[2023-01-10T09:54:22+08:00] Instance type not operational driver=qemu err="KVM support is missing (no /dev/kvm)" type=virtual-machine
INFO [2023-01-10T09:54:22+08:00] Instance type operational driver=lxc features="[]" type=container
INFO [2023-01-10T09:54:22+08:00] Initializing local database
DEBUG [2023-01-10T09:54:22+08:00] Refreshing local trusted certificate cache
INFO [2023-01-10T09:54:22+08:00] Set client certificate to server certificate fingerprint=0873aa75a432f295b901487cad44f681ffc1ac9f5e059109efd73399eef4c9b6
DEBUG [2023-01-10T09:54:22+08:00] Initializing database gateway
INFO [2023-01-10T09:54:22+08:00] Starting database node id=1 local=1 role=voter
WARNING[2023-01-10T09:54:23+08:00] Failed setting up shared mounts err="no such file or directory"
INFO [2023-01-10T09:54:23+08:00] Loading daemon configuration
INFO [2023-01-10T09:54:23+08:00] Closing socket socket="[::]:8443" type="REST API TCP socket"
INFO [2023-01-10T09:54:23+08:00] Closing socket socket=/var/snap/lxd/common/lxd/unix.socket type="REST API Unix socket"
INFO [2023-01-10T09:54:23+08:00] Closing socket socket=/var/snap/lxd/common/lxd/devlxd/sock type="devlxd socket"
ERROR [2023-01-10T09:54:23+08:00] Failed to start the daemon err="Bind network address: listen tcp :8443: bind: address already in use"
INFO [2023-01-10T09:54:23+08:00] Starting shutdown sequence signal=interrupt
DEBUG [2023-01-10T09:54:23+08:00] Cancel ongoing or future gRPC connection attempts
INFO [2023-01-10T09:54:23+08:00] Stop database gateway
INFO [2023-01-10T09:54:23+08:00] Not unmounting temporary filesystems (instances are still running)
INFO [2023-01-10T09:54:23+08:00] Daemon stopped
Error: Bind network address: listen tcp :8443: bind: address already in use
I quite puzzled why it’s giving the error of being unable to bind to tcp :8443 when nothing is showing up from lsof -i :8443 or sudo netstat -plnta | grep 8443.
I dont know that command is working, havent tried, but you can use lxd recover options for recover old storage instead of this you should backup your configs manually, please examine the link. https://linuxcontainers.org/lxd/docs/master/backup/
Regards.
That’s great, thanks. I managed to recover through the following process:
Removal via snap remove lxd
Fresh installation via snap install lxd
lxd init without creating a new storage pool (I mainly wanted it to create the network bridge and configure default settings)
lxd recover the default storage pool
Sidenote: I also had another storage pool for Docker created via a process similar to that described at this tutorial, recovery was pretty straightforward too, apart from having to figure out what path to provide (turned out to be something similar to /var/snap/lxd/common/lxd/disks/docker.img)
This whole process turned out to be fairly painless in the end, and your advice on the manual backup of configs is something that I will definitely be implementing.
Many thanks to @cemzafer and @tomp for suggestions on how to debug and recover!
As a side question: I’m curious about the usage of namespaces in LXD. I used rsync to copy out the contents of /var/snap/lxd/common/lxd and things seemed to work out, but I’m wondering if there are situations where things may not be properly backed up if files are not visible due to the separate namespaces.
Yes there are situations where that wouldn’t work.
Anything that requires a mount will not show up in /var/snap/lxd/common/lxd on the host, but would only be inside the snap’s mount namespace. For example instances have a directory in /var/snap/lxd/common/lxd but it may be empty on the host, whereas inside the snap its mounted onto a storage pool.
I have been thinking a bit about this problem and how I could avoid it in future. Since I was able to restore from my storage pools (one on an external ZFS pool, one on a loop-backed BTRFS pool), starting from a fresh installation, it seems that I must have somehow messed up my config somewhere in the database.
My LXD installation (and root installation) is on an ext4 volume. It seems to me that if the LXD installation had been on a ZFS volume with regular snapshots scheduled, the solution to this would have been as easy as rolling back /var/snap/lxd/common/lxd to a previous snapshot. Would I be right in this assumption?
Possibly, but again, that would only snapshot the contents of the ZFS dataset hosting LXD, and if you were using other storage pools that mounted ontop of that those would not be included in the snapshot.