I have an LXD container running a service with two exposed tcp ports. The service was running fine for a few days without any issues. Today I noticed the service was in an error state and when I investigated, noticed that my host OS was no longer listening on one of the ports. I confirmed this locally with ss and externally with nmap. I checked the container configuration and both proxy devices were listed. I then issued an lxc restart command on the container and after it restarted, both ports are listening again and the service is working. Any ideas what could have happened or where I can look to diagnose this?
Proxy might have crashed for some reason. There are log files for it in /var/snap/lxd/common/lxd/logs/NAME which you can look at.
What version of LXD was that?
I’m running 3.15 on Ubuntu 18.04. I don’t have /var/snap/lxd/common/lxd/logs/, the only thing in /var/snap/lxd/common/lxd/ is a unix.socket. I’ve started looking through stuff in /var/log/lxd but I don’t see anything describing an error. I found a log file for the specific proxy device but it just shows “Starting to proxy” followed by a ton of errors without timestamps of ips trying to connect to the port after it closed.
Oh, you may be using a mix of snap and deb maybe?
Can you show:
- lxc version
- dpkg -l | grep lxd
If I am it’s not on purpose. This is a vps instance image so maybe lxd was installed by default and before I did my snap install.
root@li1014-78:~# lxc version Client version: 3.0.3 Server version: 3.0.3 root@li1014-78:~# dpkg -l | grep lxd ii lxd 3.0.3-0ubuntu1~18.04.1 amd64 Container hypervisor based on LXC - daemon ii lxd-client 3.0.3-0ubuntu1~18.04.1 amd64 Container hypervisor based on LXC - client
Right, so you’re actually running LXD 3.0.3 + have the snap installed, which leads to pretty confusing behaviors
I’d recommend moving your data over to the snap and removing the debs.
This can all be done with
lxd.migrate which will interactively let you migrate the data across and cleanup the deb.
Thanks for the help. I wish Ubuntu would alert me if I’m trying to install the snap of a package that is already installed via apt.
Yeah, we don’t have detection working that way around. We do have logic which prevents you from using the snap if the deb is still around though.