Copying a container with a static IP to a different host

1

I have setup a container using a static IP address and I am copying/migrating to a different host.

$ lxc copy a1 server2:a2

When copying the instance to a different server, despite the instance now having a new IP address, both devices.eth0.ipv4.address and expanded_devices.eth0.ipv4.address report the original IP address on the other server 10.247.51.242, but the network.eth0.addresses.0.ipv4.address is the newly allocated IP 10.135.210.168

Is this correct that the copy of the instance devices show a different IP address that what is assigned?

Also proxy devices using the nat=true retain the original IP address when copying, I guess I need to just delete the proxies and re-create these with the new IP address?

"my-proxy": {
    "connect": "tcp:10.247.51.242:80",
    "listen": "tcp:192.168.1.100:8080",
    "nat": "true",
    "type": "proxy"
}

Any guidance on the best way to deal with these issues would be greatly appreciated. Thanks.

2

Please can you show the output of lxc config show <instance> and lxc config show <instance> --expanded for both the original and the new instance. Thanks

3

lxc config show a1

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Alpine 3.11 amd64 (20200703_13:00)
  image.os: Alpine
  image.release: "3.11"
  image.serial: "20200703_13:00"
  image.type: squashfs
  limits.cpu: "1"
  limits.memory: 1GB
  volatile.base_image: b769f70a789cff58d63d2f34d917a2d6c9b185167d1c659facd48124f4aafb05
  volatile.eth0.hwaddr: 00:16:3e:8a:30:c5
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: STOPPED
  volatile.uuid: 5bf70fa7-021f-4110-8281-b59625d7c562
devices:
  eth0:
    ipv4.address: 10.247.51.242
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  my-proxy:
    connect: tcp:10.247.51.242:80
    listen: tcp:192.168.1.150:8080
    nat: "true"
    type: proxy
  root:
    path: /
    pool: default
    size: 5GB
    type: disk
ephemeral: false
profiles:
- my-default
stateful: false
description: ""

lxc config show a2

architecture: x86_64
config:
  image.architecture: amd64
  image.description: Alpine 3.11 amd64 (20200703_13:00)
  image.os: Alpine
  image.release: "3.11"
  image.serial: "20200703_13:00"
  image.type: squashfs
  limits.cpu: "1"
  limits.memory: 1GB
  volatile.base_image: b769f70a789cff58d63d2f34d917a2d6c9b185167d1c659facd48124f4aafb05
  volatile.eth0.host_name: veth8a8c20cc
  volatile.eth0.hwaddr: 00:16:3e:83:bb:a2
  volatile.idmap.base: "0"
  volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
  volatile.last_state.power: RUNNING
devices:
  eth0:
    ipv4.address: 10.247.51.242
    name: eth0
    nictype: bridged
    parent: lxdbr0
    type: nic
  my-proxy:
    connect: tcp:10.247.51.242:80
    listen: tcp:192.168.1.150:8080
    nat: "true"
    type: proxy
  root:
    path: /
    pool: default
    size: 5GB
    type: disk
ephemeral: false
profiles:
- my-default
stateful: false
description: ""

If i run lxc info a2

Ips:
  eth0:	inet	10.135.210.47	veth8a8c20cc
  eth0:	inet6	fd42:242e:79f1:8f9c:216:3eff:fe83:bba2	veth8a8c20cc
  eth0:	inet6	fe80::216:3eff:fe83:bba2	veth8a8c20cc
  lo:	inet	127.0.0.1
  lo:	inet6	::1
4

Can you show the output of lxc network show lxdbr0 on both hosts please?

Also can you show the outputs of the previous command with and without the --expanded flag as requested so I can see whats coming from the profile and the instance itself.

5 
config:
  ipv4.address: 10.247.51.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:9d94:dbb9:4e8b::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/a1
- /1.0/instances/instance-test
- /1.0/instances/custom-app
- /1.0/profiles/default
- /1.0/profiles/custom-nat
managed: true
status: Created
locations:
- none

config:
  ipv4.address: 10.135.210.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:242e:79f1:8f9c::1/64
  ipv6.nat: "true"
description: ""
name: lxdbr0
type: bridge
used_by:
- /1.0/instances/a2
- /1.0/instances/custom-app
- /1.0/profiles/default
- /1.0/profiles/custom-nat
managed: true
status: Created
locations:
- none
6

I’ve just tried that here on LXD 4.9 and it seems to work fine:

Host v1:

lxc network show lxdbr0
config:
  ipv4.address: 10.236.91.1/24
  ipv4.nat: "true"
  ipv6.address: fd42:9965:d291:70::1/64
  ipv6.nat: "true"

lxc init images:ubuntu/focal c1
lxc config device override c1 eth0 ipv4.address=10.236.91.10
lxc start c1

lxc remote add v2 10.87.156.79

Host v2:

lxc network set lxdbr0 ipv4.address=10.236.91.1/24 ipv6.address=fd42:9965:d291:70::1/64

Host v1:

lxc stop c1
lxc copy c1 v2:c2

Host v2:

sudo cat /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/c2 
00:16:3e:20:b7:1b,10.236.91.10,c2

lxc start c2
lxc ls c2
+------+---------+---------------------+---------------------------------------------+-----------+-----------+
| NAME |  STATE  |        IPV4         |                    IPV6                     |   TYPE    | SNAPSHOTS |
+------+---------+---------------------+---------------------------------------------+-----------+-----------+
| c2   | RUNNING | 10.236.91.10 (eth0) | fd42:9965:d291:70:216:3eff:fe20:b71b (eth0) | CONTAINER | 0         |
+------+---------+---------------------+---------------------------------------------+-----------+-----------+
7

Oh I see the issue, the subnets of your lxdbr0 networks are different.

Can you show the output of the command below on the target host, to see if the IP has been set correctly and dnsmasq is just ignoring it because its in the wrong subnet:

sudo cat /var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts/a2
8

The lxc config show c2 on the target, shows me incorrect information.

9

Yes you are correct subnets are different.

00:16:3e:83:bb:a2,10.247.51.242,a2
10

You showed earlier that the config is correct and your latest post shows the static IP was correctly written into the dnsmasq config, but the lxc info <instance> command is showing different IPs, this is because dnsmasq has ignored the static config as its part of a different subnet and instead allocated a dynamic IP to the container.

Your lxdbr0 subnets need to match.

11

Yes that is correct and expected, as the proxy devices allow you to specify an external IP (which might be the same on both hosts). You can just modify the device on the copy using:

lxc config device set <instance> <device> listen=...

For the proxy’s connect setting, you can specify 0.0.0.0 as the IP and it should take the current container’s IP as the target automatically.

12

So basically when working with multiple LXD hosts, I should ensure that they are all configured with the same subnets.

13

Yes thats correct, otherwise if you specify a static IP in a subnet that isn’t the same as the host’s subnet then it will fail to allocate.

14

Thanks for your help.

1 Like