Lxc copy --refresh and mac address

Hi

What ways do you use refreshing the destination container (web server) when the destination LXD is running containers for haproxy and web server?

I have the haproxy and related web server environment configured to use the default lxdbr0.

I stopped first the destination container (web server ) and from my development instance ran this:

lxc copy web1 server2:web1 --refresh

but if the haproxy is running it can’t find anymore the web1.lxd after starting the web1 (running web site). If I reboot the haproxy it’s same, still can’t find the destination web1.lxd.

But if I do from beginning same copy first stopping destination web1 and copying but without refresh like this:

lxc copy web1 server2:web1

After starting web1 the haproxy finds the new container with the same mac it’s used to for web1.lxd).

Can I somehow avoid the above issue with the destination network lxdbr0?

Thanks

I’m not entirely clear what the issue you are experience is?

Is there a specific error message or test that is failing?

Can you provide a full reproducer example please. Thanks

lxc info server2:haproxy
Name: haproxy
Status: RUNNING
Type: container
Architecture: x86_64
PID: 247742
Created: 2021/04/24 18:03 UTC
Last Used: 2021/08/16 07:39 UTC

Resources:
Processes: 26
Disk usage:
root: 2.33GiB
CPU usage:
CPU usage (in seconds): 6
Memory usage:
Memory (current): 163.06MiB
Memory (peak): 188.24MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: veth4297bea8
MAC address: 00:16:3e:a2:cf:d2
MTU: 1500
Bytes received: 13.44MB
Bytes sent: 13.54MB
Packets received: 13710
Packets sent: 13884
IP addresses:
inet: 10.94.250.219/24 (global)
inet6: fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2/64 (global)
inet6: fe80::216:3eff:fea2:cfd2/64 (link)

lxc info server2:web1
Name: web1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 313776
Created: 2021/08/16 07:46 UTC
Last Used: 2021/08/16 08:30 UTC

Resources:
Processes: 84
Disk usage:
root: 1.64GiB
CPU usage:
CPU usage (in seconds): 71
Memory usage:
Memory (current): 206.49MiB
Memory (peak): 329.13MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: veth2de216cf
MAC address: 00:16:3e:53:ab:12
MTU: 1500
Bytes received: 2.25MB
Bytes sent: 20.09MB
Packets received: 14202
Packets sent: 13902
IP addresses:
inet: 10.94.250.229/24 (global)
inet6: fd42:3bd5:1f49:ee2e:216:3eff:fe53:ab12/64 (global)
inet6: fe80::216:3eff:fe53:ab12/64 (link)

this same web1 which I have in my LXD development instance I’m trying to copy it to the server2: using command lxc copy web1 server2:web1 --refresh:

lxc info web1
Name: web1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 116935
Created: 2021/05/06 08:13 UTC
Last Used: 2021/08/16 09:39 UTC

Resources:
Processes: 100
Disk usage:
root: 1.78GiB
CPU usage:
CPU usage (in seconds): 2
Memory usage:
Memory (current): 173.01MiB
Memory (peak): 175.66MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: vethf1e84c9f
MAC address: 00:16:3e:16:4e:e5
MTU: 1500
Bytes received: 2.38kB
Bytes sent: 2.66kB
Packets received: 18
Packets sent: 24
IP addresses:
inet: 10.240.195.229/24 (global)
inet6: fd42:12c6:7039:90f0:216:3eff:fe16:4ee5/64 (global)
inet6: fe80::216:3eff:fe16:4ee5/64 (link)

When I lxc copy web1 server2:web1 --refresh and start the destination server2:web1 - the other container server2:haproxy still running in same destination lxdbr0 network doesn’t anymore find the host web1.lxd after starting it - I expect to be the same host, just refreshed/synced and running on server:

lxc list server2:
±---------------±--------±---------------------±----------------------------------------------±----------------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±---------------±--------±---------------------±----------------------------------------------±----------------±----------+

| haproxy | RUNNING | 10.94.250.219 (eth0) | fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2 (eth0) | CONTAINER | 180 |
±---------------±--------±---------------------±----------------------------------------------±----------------±----------+

| web1 | RUNNING | 10.94.250.229 (eth0) | fd42:3bd5:1f49:ee2e:216:3eff:fe16:4ee5 (eth0) | CONTAINER | 1 |
±---------------±--------±---------------------±----------------------------------------------±----------------±----------+

lxc exec server2:haproxy – bash
root@haproxy:~# ping web1.lxd
PING web1.lxd(web1.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fe64:1fe0)) 56 data bytes
From haproxy.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2) icmp_seq=1 Destination unreachable: Address unreachable
From haproxy.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2) icmp_seq=2 Destination unreachable: Address unreachable
From haproxy.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2) icmp_seq=3 Destination unreachable: Address unreachable

How can I workaround or is the lxc copy --refresh option intended more like for backups ?

Hrm, I’m still not clear what you are saying I’m afraid.

I think you are saying that when you copy a container from one host to the other using refresh mode then there are MAC address conflicts?

But its not clear that both the containers are on the same L2 network, please can you show the network config using lxc network show <network> from both servers.

Also can you show the lxc config show <instance> --expanded for both the source and copied containers.

Thanks

The issue is in process where both hosts containers use their own lxdbr0 networks.

copying a container from one host to other host using refresh mode results in new MAC address when starting it in the destination lxdbr0 network. (The copied instance MAC is different from the to be refreshed old web1 instance originally in destination host) The networks where the containers are connected to are on both hosts in their own default lxdbr0 network - that’s the initial network setup already when installing/init lxd

Please can you advise why the MAC address changing when connected to a different private lxdbr0 is a problem?

Can you show the output of lxc network show lxdbr0 on both hosts please?

I am unclear why a different MAC address on a different network is a problem?

I didn’t first think the MAC addresses are a problem but don’t know because it doesn’t seem to change when I don’t use the refresh mode. I got from your previous reply a thought that the issue is with the MAC network - but the containers are configured to use the default private lxdbr0 on both hosts.

I try to repeat the “problem” with another container from my source:

lxc config show web2
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20210415)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: “20210415”
image.type: squashfs
image.version: “18.04”
volatile.apply_template: copy
volatile.base_image: cc14c502ccc1fe07f322649d9610fb240ed95866db72605afa33010cadfe9c22
volatile.eth0.hwaddr: 00:16:3e:e7:f7:20
volatile.idmap.base: “0”
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.power: STOPPED
devices: {}
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”

lxc network show lxdbr0
config:
ipv4.address: 10.240.195.1/24
ipv4.nat: “true”
ipv6.address: fd42:12c6:7039:90f0::1/64
ipv6.nat: “true”
description: “”
name: lxdbr0
type: bridge
used_by:

  • /1.0/instances/web1
  • /1.0/instances/web2
  • /1.0/profiles/default
  • /1.0/profiles/haprofile
    managed: true
    status: Created
    locations:
  • none

lxc start web2

lxc info web2
Name: web2
Status: RUNNING
Type: container
Architecture: x86_64
PID: 16098
Created: 2021/05/06 08:13 UTC
Last Used: 2021/08/17 17:54 UTC

Resources:
Processes: 24
Disk usage:
root: 989.07MiB
CPU usage:
CPU usage (in seconds): 10
Memory usage:
Memory (current): 102.95MiB
Memory (peak): 159.90MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: veth044515d1
MAC address: 00:16:3e:e7:f7:20
MTU: 1500
Bytes received: 900.62kB
Bytes sent: 44.22kB
Packets received: 717
Packets sent: 601
IP addresses:
inet: 10.240.195.188/24 (global)
inet6: fd42:12c6:7039:90f0:216:3eff:fee7:f720/64 (global)
inet6: fe80::216:3eff:fee7:f720/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 1.46kB
Bytes sent: 1.46kB
Packets received: 16
Packets sent: 16
IP addresses:
inet: 127.0.0.1/8 (local)
inet6: ::1/128 (local)

lxc stop web2
lxc copy web2 server2:web2 --verbose --mode=push

lxc config show server2:web2
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20210415)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: “20210415”
image.type: squashfs
image.version: “18.04”
volatile.apply_template: copy
volatile.base_image: cc14c502ccc1fe07f322649d9610fb240ed95866db72605afa33010cadfe9c22
volatile.eth0.hwaddr: 00:16:3e:70:17:3d
volatile.idmap.base: “0”
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
devices: {}
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”

lxc network show server2:lxdbr0
config:
ipv4.address: 10.94.250.1/24
ipv4.nat: “true”
ipv6.address: fd42:3bd5:1f49:ee2e::1/64
ipv6.nat: “true”
description: “”
name: lxdbr0
type: bridge
used_by:

  • /1.0/instances/haproxy
  • /1.0/instances/web1
  • /1.0/instances/web2
  • /1.0/profiles/default
  • /1.0/profiles/haprofile
    managed: true
    status: Created
    locations:
  • none

lxc start server2:web2
lxc exec server2:haproxy – bash
root@haproxy:~# ping web2.lxd
PING web2.lxd(web2.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fe70:173d)) 56 data bytes
64 bytes from web2.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fe70:173d): icmp_seq=1 ttl=64 time=0.117 ms
64 bytes from web2.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fe70:173d): icmp_seq=2 ttl=64 time=0.079 ms
^C
— web2.lxd ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.079/0.098/0.117/0.019 ms

This is what we do now clearly works but let’s try again the same lxc copy in refresh mode:

exit
exit
lxc stop server2:web2
lxc start web2
lxc stop web2
lxc copy web2 server2web2 --verbose --refresh --mode=push

lxc config show server2:web2
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20210415)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: “20210415”
image.type: squashfs
image.version: “18.04”
volatile.base_image: cc14c502ccc1fe07f322649d9610fb240ed95866db72605afa33010cadfe9c22
volatile.eth0.hwaddr: 00:16:3e:e7:f7:20
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.uuid: 8ed67416-02c2-4ce0-b2c8-23e40ece7f8d
devices: {}
ephemeral: false
profiles:

  • default
    stateful: false
    description: “”
    lxc network show server2:lxdbr0
    config:
    ipv4.address: 10.94.250.1/24
    ipv4.nat: “true”
    ipv6.address: fd42:3bd5:1f49:ee2e::1/64
    ipv6.nat: “true”
    description: “”
    name: lxdbr0
    type: bridge
    used_by:
  • /1.0/instances/haproxy
  • /1.0/instances/web1
  • /1.0/instances/web2
  • /1.0/profiles/default
  • /1.0/profiles/haprofile
    managed: true
    status: Created
    locations:
  • none

lxc start server2:web2
lxc exec server2:haproxy – bash
root@haproxy:~# ping web2.lxd
PING web2.lxd (10.94.250.225) 56(84) bytes of data.
From haproxy.lxd (10.94.250.219) icmp_seq=1 Destination Host Unreachable
From haproxy.lxd (10.94.250.219) icmp_seq=2 Destination Host Unreachable
From haproxy.lxd (10.94.250.219) icmp_seq=3 Destination Host Unreachable
^C
— web2.lxd ping statistics —
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4079ms
pipe 4
root@haproxy:~#

I did not check the ipv4 address for server2:web2 after the first time run (after copying first without refresh mode)

lxc info server2:web2
Name: web2
Status: RUNNING
Type: container
Architecture: x86_64
PID: 1620536
Created: 2021/08/17 18:12 UTC
Last Used: 2021/08/17 18:35 UTC

Resources:
Processes: 24
Disk usage:
root: 914.71MiB
CPU usage:
CPU usage (in seconds): 4
Memory usage:
Memory (current): 130.04MiB
Memory (peak): 176.66MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: vethfc0beeb0
MAC address: 00:16:3e:e7:f7:20
MTU: 1500
Bytes received: 1.77kB
Bytes sent: 2.12kB
Packets received: 17
Packets sent: 20
IP addresses:
inet: 10.94.250.188/24 (global)
inet6: fd42:3bd5:1f49:ee2e:216:3eff:fee7:f720/64 (global)
inet6: fe80::216:3eff:fee7:f720/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 276B
Bytes sent: 276B
Packets received: 4
Packets sent: 4
IP addresses:
inet: 127.0.0.1/8 (local)
inet6: ::1/128 (local)

One more detail from the server2:haproxy it is running now all the time, and it is configured with the server2:haprofile to use static ipv4.address. (forwarding from a host device haproxy related incoming http traffic):

lxc profile show server2:default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: lxd-pool
type: disk
name: default
used_by:

  • /1.0/instances/haproxy
  • /1.0/instances/web1
  • /1.0/instances/web2

lxc profile show server2:haprofile
config: {}
description: Bridged networking LXD profile
devices:
eth0:
ipv4.address: 10.94.250.219
nictype: bridged
parent: lxdbr0
type: nic
name: haprofile
used_by:

  • /1.0/instances/haproxy

I did wait some minutes this time and after some time the ping to the host web2.lxd start to work.

lxc exec server2:haproxy-- bash
root@haproxy:~# ping web2.lxd
PING web2.lxd (10.94.250.188) 56(84) bytes of data.
64 bytes from web2.lxd (10.94.250.188): icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from web2.lxd (10.94.250.188): icmp_seq=2 ttl=64 time=0.065 ms
^C
— web2.lxd ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1010ms
rtt min/avg/max/mdev = 0.065/0.077/0.089/0.012 ms

what’s the ipv4/ipv6 related difference when using lxc copy without refresh mode ?
I have had this small issue getting the private connection work between the haproxy and web1/web2 now on 3-4 different hosts after using --refresh mode exactly like in the last example here with web2

Let me know if there’s need to have some specific syslog from destination host.

Hi,

Sorry for the delay in replying to this.

So I’ve had a chance now to investigate this and I believe I have an answer for you.

When performing a copy of an instance without the --refresh flag the LXD client lxc strips the volatile.eth0.hwaddr setting, whereas with the --refresh flag the lxc client will remove the volatile.eth0.hwaddr setting, causing a new one to be generated with a different MAC address.

This happens even if there is already an existing instance with a different MAC address.

E.g.

lxc copy c1 c2 --refresh # c2 will have same volatile.eth0.hwaddr as c1.
lxc delete c2
lxc copy c1 c2 # c2 will have different volatile.eth0.hwaddr than c1.
lxc copy c1 c2 --refresh # c2 will now have same volatile.eth0.hwaddr as c1.

The change of MAC address between the initial copy of lxc copy c1 c2 without the --refresh flag is probably what is causing the pause of communication until the target server detects the instance’s MAC has changed.

Assuming this is the desired behaviour (CC @stgraber), then as the two servers have private bridges on separate networks and using the same MAC address on both won’t be an issue, then you can just to --refresh from the first copy and ensure the MAC address won’t change.

Yeah, from what I remember, we have lxc copy strip volatile whereas lxc move doesn’t. The reason being that we don’t want to cause conflicts when an instance is duplicated.

I believe it’s intentional that --refresh gets us the same behavior as lxc move as its goal is to perfectly replicate an instance as used for backups and the like.

1 Like

Thank you,

lxc copy (without --refresh) has not caused problems (delays detecting MAC changed.)

I was expecting it somehow that the way I was using lxc copy --refresh was not ideal for starting the same instance (refreshed duplicate) in the destination server on separate network. I

1 Like