SOLVED: Upgraded to LXD 3.21, and now my container won't start

I have a container that’s been in production for months. However after upgrading to LXD 3.21 (along with other system upgrades), the container will no longer start. (I unfortunately had to run updates on the host system; this wasn’t a voluntary thing.) I get this error message:

[pgoetz@erap-atx pkg]$ lxc start atom
Error: Common start logic: Failed to start device "atomport8084": Invalid value for device option "connect": Not an IP address: localhost
Try `lxc info --show-log atom` for more info

[pgoetz@erap-atx pkg]$ lxc info --show-log atom
Name: atom
Location: none
Remote: unix://
Architecture: x86_64
Created: 2019/06/05 19:06 UTC
Status: Stopped
Type: container
Profiles: default
Snapshots:
  atom-catalog-new-install-2020-01-29 (taken at 2020/01/29 18:50 UTC) (stateless)

Log:

lxc atom 20200224195502.269 WARN     conf - conf.c:lxc_setup_devpts:1641 - Invalid argument - Failed to unmount old devpts instance
lxc atom 20200224230242.332 WARN     network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 7

The proxy it’s complaining about in the initial error message was configured like this:

lxc config device add atom atomport8080 proxy listen=tcp:0.0.0.0:8080 connect=tcp:localhost:80

Do I need to redo this with an IP address?

Any thoughts on what is going on? This looks a lot like

This is the issue,

Error: Common start logic: Failed to start device "atomport8084": Invalid value for device option "connect": Not an IP address: localhost

That is, you have a proxy device that mentions a hostname while you should have an IP address.
Initially, LXD was accepting hostnames but later disallowed them. That change was sometime last year. It is amazing that the container did not restart during so many LXD upgrades, meaning that the new check against hostnames did not kick in.

You can edit the container configuration to have the IP address (127.0.0.1) instead of localhost.

More on that, https://blog.simos.info/how-to-use-the-lxd-proxy-device-to-map-ports-between-the-host-and-the-containers/

1 Like

Thanks, Simos! I was just updating the ticket with this information, so good timing!

BTW, your blog link isn’t loading – is your server down? I can’t ping it, either.

Thanks for reporting. The server was down and I rebooted it.
Now it is working again.

Using 127.0.0.1 doesn’t appear to work, either:

[pgoetz@erap-atx pkg]$ lxc config device add atom atomport8080 proxy listen=tcp:0.0.0.0:8080 connect=tcp:127.0.0.1:80
Error: Invalid devices: Device validation failed "atomport8084": Invalid value for device option "connect": Not an IP address: localhost

Also, very oddly, when I try to delete the old device it can’t be found:

[pgoetz@erap-atx pkg]$ lxc config device remove atom atomport8080
Error: The device doesn't exist

I can’t figure out where the container configuration files are. Or I should say, they appear to be in

/var/lib/lxd/containers

but there’s no mention in the atom/metadata.yaml file of any proxy port, and the

/var/lib/lxd/devices/atom

folder is empty.

You are using LXD 3.21 and the above configuration locations are for the DEB package of LXD. That is, you are using the snap package of LXD (latest stable: 3.21) and the location of the files is now at /var/snap/lxd/....

To verify which version of LXD you are using, you can lxc --version.

A LXD container may have configuration applied directly to the container, or configuration that was derived from a profile.
To show the full configuration of a container, run the following,

lxc config show atom --expanded

Note that if your container configuration has two proxy devices that specify localhost (instead of 127.0.0.1), then if you are to fix them or even remove them, you have to do both at the same time. That means, you have to change them both, because the removal process can remove one proxy device at a time (which cannot be done because after the first removal, the configuration would still be invalid due to the other mention of localhost).

1 Like

The host OS is Arch linux, and the lxd package is the tar.gz file installed directly from github, so in particular I’m not using snaps:

[pgoetz@erap-atx ~]$ ls /var/snap
ls: cannot access '/var/snap': No such file or directory

However, I’m getting more confused. There appear to be yaml files here:

/var/lib/lxd/storage-pools/default/containers/atom
[root@erap-atx atom]# ls
backup.yaml  metadata.yaml  rootfs  template

The backup.yaml file appears to have the right stuff in it, but metadata.yaml just has boilerplate unrelated to the specific built container. I only have one proxy associated with this container, which is displayed when I run
lxc config show atom --expanded

[root@erap-atx atom]# lxc config show atom --expanded
architecture: x86_64
config:
  image.architecture: amd64
  image.description: ubuntu 18.04 LTS amd64 (release) (20190604)
  image.label: release
  image.os: ubuntu
  image.release: bionic
  image.serial: "20190604"
  image.version: "18.04"
  security.privileged: "true"
  volatile.base_image: c234ecee3baaee25db84af8e3565347e948bfceb3bf7c820bb1ce95adcffeaa8
  volatile.eth0.host_name: vethb7b46128
  volatile.eth0.hwaddr: 00:16:3e:03:f6:fd
  volatile.eth0.name: eth0
  volatile.idmap.base: "0"
  volatile.idmap.current: '[]'
  volatile.idmap.next: '[]'
  volatile.last_state.idmap: '[]'
  volatile.last_state.power: STOPPED
devices:
  atomport8084:
    connect: tcp:localhost:80
    listen: tcp:0.0.0.0:8084
    type: proxy
  eth0:
    ipv4.address: 10.248.83.4
    nictype: bridged
    parent: lxdbr0
    type: nic
  root:
    path: /
    pool: default
    type: disk
ephemeral: false
profiles:
- default
stateful: false
description: ""

So the yaml file must be somewhere, I just can’t find it.

But in any case I mistyped the proxy removal last time, which is why it failed – I had 8080 instead of 8084. This time it worked, so this sequence of commands resolved the issue:

[pgoetz@erap-atx lxd]$ lxc config device remove atom atomport8084
[pgoetz@erap-atx lxd]$ lxc config device add atom atomport8084 proxy listen=tcp:0.0.0.0:8084 connect=tcp:127.0.0.1:80

I’m going to mark this as solved. Thanks so much, Simos!