Lxc copy --refresh and mac address

vsuojanen · August 16, 2021, 8:58am

Hi

What ways do you use refreshing the destination container (web server) when the destination LXD is running containers for haproxy and web server?

I have the haproxy and related web server environment configured to use the default lxdbr0.

I stopped first the destination container (web server ) and from my development instance ran this:

lxc copy web1 server2:web1 --refresh

but if the haproxy is running it can’t find anymore the web1.lxd after starting the web1 (running web site). If I reboot the haproxy it’s same, still can’t find the destination web1.lxd.

But if I do from beginning same copy first stopping destination web1 and copying but without refresh like this:

lxc copy web1 server2:web1

After starting web1 the haproxy finds the new container with the same mac it’s used to for web1.lxd).

Can I somehow avoid the above issue with the destination network lxdbr0?

Thanks

tomp · August 16, 2021, 9:09am

I’m not entirely clear what the issue you are experience is?

Is there a specific error message or test that is failing?

Can you provide a full reproducer example please. Thanks

vsuojanen · August 16, 2021, 9:54am

lxc info server2:haproxy
Name: haproxy
Status: RUNNING
Type: container
Architecture: x86_64
PID: 247742
Created: 2021/04/24 18:03 UTC
Last Used: 2021/08/16 07:39 UTC

Resources:
Processes: 26
Disk usage:
root: 2.33GiB
CPU usage:
CPU usage (in seconds): 6
Memory usage:
Memory (current): 163.06MiB
Memory (peak): 188.24MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: veth4297bea8
MAC address: 00:16:3e:a2:cf:d2
MTU: 1500
Bytes received: 13.44MB
Bytes sent: 13.54MB
Packets received: 13710
Packets sent: 13884
IP addresses:
inet: 10.94.250.219/24 (global)
inet6: fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2/64 (global)
inet6: fe80::216:3eff:fea2:cfd2/64 (link)

lxc info server2:web1
Name: web1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 313776
Created: 2021/08/16 07:46 UTC
Last Used: 2021/08/16 08:30 UTC

Resources:
Processes: 84
Disk usage:
root: 1.64GiB
CPU usage:
CPU usage (in seconds): 71
Memory usage:
Memory (current): 206.49MiB
Memory (peak): 329.13MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: veth2de216cf
MAC address: 00:16:3e:53:ab:12
MTU: 1500
Bytes received: 2.25MB
Bytes sent: 20.09MB
Packets received: 14202
Packets sent: 13902
IP addresses:
inet: 10.94.250.229/24 (global)
inet6: fd42:3bd5:1f49:ee2e:216:3eff:fe53:ab12/64 (global)
inet6: fe80::216:3eff:fe53:ab12/64 (link)

this same web1 which I have in my LXD development instance I’m trying to copy it to the server2: using command lxc copy web1 server2:web1 --refresh:

lxc info web1
Name: web1
Status: RUNNING
Type: container
Architecture: x86_64
PID: 116935
Created: 2021/05/06 08:13 UTC
Last Used: 2021/08/16 09:39 UTC

Resources:
Processes: 100
Disk usage:
root: 1.78GiB
CPU usage:
CPU usage (in seconds): 2
Memory usage:
Memory (current): 173.01MiB
Memory (peak): 175.66MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: vethf1e84c9f
MAC address: 00:16:3e:16:4e:e5
MTU: 1500
Bytes received: 2.38kB
Bytes sent: 2.66kB
Packets received: 18
Packets sent: 24
IP addresses:
inet: 10.240.195.229/24 (global)
inet6: fd42:12c6:7039:90f0:216:3eff:fe16:4ee5/64 (global)
inet6: fe80::216:3eff:fe16:4ee5/64 (link)

When I lxc copy web1 server2:web1 --refresh and start the destination server2:web1 - the other container server2:haproxy still running in same destination lxdbr0 network doesn’t anymore find the host web1.lxd after starting it - I expect to be the same host, just refreshed/synced and running on server:

lxc list server2:
±---------------±--------±---------------------±----------------------------------------------±----------------±----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
±---------------±--------±---------------------±----------------------------------------------±----------------±----------+

| haproxy | RUNNING | 10.94.250.219 (eth0) | fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2 (eth0) | CONTAINER | 180 |
±---------------±--------±---------------------±----------------------------------------------±----------------±----------+

| web1 | RUNNING | 10.94.250.229 (eth0) | fd42:3bd5:1f49:ee2e:216:3eff:fe16:4ee5 (eth0) | CONTAINER | 1 |
±---------------±--------±---------------------±----------------------------------------------±----------------±----------+

lxc exec server2:haproxy – bash
root@haproxy:~# ping web1.lxd
PING web1.lxd(web1.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fe64:1fe0)) 56 data bytes
From haproxy.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2) icmp_seq=1 Destination unreachable: Address unreachable
From haproxy.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2) icmp_seq=2 Destination unreachable: Address unreachable
From haproxy.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fea2:cfd2) icmp_seq=3 Destination unreachable: Address unreachable

How can I workaround or is the lxc copy --refresh option intended more like for backups ?

tomp · August 16, 2021, 10:21am

Hrm, I’m still not clear what you are saying I’m afraid.

I think you are saying that when you copy a container from one host to the other using refresh mode then there are MAC address conflicts?

But its not clear that both the containers are on the same L2 network, please can you show the network config using lxc network show <network> from both servers.

Also can you show the lxc config show <instance> --expanded for both the source and copied containers.

Thanks

vsuojanen · August 16, 2021, 1:01pm

The issue is in process where both hosts containers use their own lxdbr0 networks.

copying a container from one host to other host using refresh mode results in new MAC address when starting it in the destination lxdbr0 network. (The copied instance MAC is different from the to be refreshed old web1 instance originally in destination host) The networks where the containers are connected to are on both hosts in their own default lxdbr0 network - that’s the initial network setup already when installing/init lxd

tomp · August 17, 2021, 2:21pm

Please can you advise why the MAC address changing when connected to a different private lxdbr0 is a problem?

Can you show the output of lxc network show lxdbr0 on both hosts please?

I am unclear why a different MAC address on a different network is a problem?

vsuojanen · August 17, 2021, 7:10pm

I didn’t first think the MAC addresses are a problem but don’t know because it doesn’t seem to change when I don’t use the refresh mode. I got from your previous reply a thought that the issue is with the MAC network - but the containers are configured to use the default private lxdbr0 on both hosts.

I try to repeat the “problem” with another container from my source:

lxc config show web2
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20210415)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: “20210415”
image.type: squashfs
image.version: “18.04”
volatile.apply_template: copy
volatile.base_image: cc14c502ccc1fe07f322649d9610fb240ed95866db72605afa33010cadfe9c22
volatile.eth0.hwaddr: 00:16:3e:e7:f7:20
volatile.idmap.base: “0”
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.power: STOPPED
devices: {}
ephemeral: false
profiles:

default
stateful: false
description: “”

lxc network show lxdbr0
config:
ipv4.address: 10.240.195.1/24
ipv4.nat: “true”
ipv6.address: fd42:12c6:7039:90f0::1/64
ipv6.nat: “true”
description: “”
name: lxdbr0
type: bridge
used_by:

/1.0/instances/web1

/1.0/instances/web2

/1.0/profiles/default

/1.0/profiles/haprofile
managed: true
status: Created
locations:

none

lxc start web2

lxc info web2
Name: web2
Status: RUNNING
Type: container
Architecture: x86_64
PID: 16098
Created: 2021/05/06 08:13 UTC
Last Used: 2021/08/17 17:54 UTC

Resources:
Processes: 24
Disk usage:
root: 989.07MiB
CPU usage:
CPU usage (in seconds): 10
Memory usage:
Memory (current): 102.95MiB
Memory (peak): 159.90MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: veth044515d1
MAC address: 00:16:3e:e7:f7:20
MTU: 1500
Bytes received: 900.62kB
Bytes sent: 44.22kB
Packets received: 717
Packets sent: 601
IP addresses:
inet: 10.240.195.188/24 (global)
inet6: fd42:12c6:7039:90f0:216:3eff:fee7:f720/64 (global)
inet6: fe80::216:3eff:fee7:f720/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 1.46kB
Bytes sent: 1.46kB
Packets received: 16
Packets sent: 16
IP addresses:
inet: 127.0.0.1/8 (local)
inet6: ::1/128 (local)

lxc stop web2
lxc copy web2 server2:web2 --verbose --mode=push

lxc config show server2:web2
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20210415)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: “20210415”
image.type: squashfs
image.version: “18.04”
volatile.apply_template: copy
volatile.base_image: cc14c502ccc1fe07f322649d9610fb240ed95866db72605afa33010cadfe9c22
volatile.eth0.hwaddr: 00:16:3e:70:17:3d
volatile.idmap.base: “0”
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
devices: {}
ephemeral: false
profiles:

default
stateful: false
description: “”

lxc network show server2:lxdbr0
config:
ipv4.address: 10.94.250.1/24
ipv4.nat: “true”
ipv6.address: fd42:3bd5:1f49:ee2e::1/64
ipv6.nat: “true”
description: “”
name: lxdbr0
type: bridge
used_by:

/1.0/instances/haproxy

/1.0/instances/web1

/1.0/instances/web2

/1.0/profiles/default

/1.0/profiles/haprofile
managed: true
status: Created
locations:

none

lxc start server2:web2
lxc exec server2:haproxy – bash
root@haproxy:~# ping web2.lxd
PING web2.lxd(web2.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fe70:173d)) 56 data bytes
64 bytes from web2.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fe70:173d): icmp_seq=1 ttl=64 time=0.117 ms
64 bytes from web2.lxd (fd42:3bd5:1f49:ee2e:216:3eff:fe70:173d): icmp_seq=2 ttl=64 time=0.079 ms
^C
— web2.lxd ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.079/0.098/0.117/0.019 ms

This is what we do now clearly works but let’s try again the same lxc copy in refresh mode:

exit
exit
lxc stop server2:web2
lxc start web2
lxc stop web2
lxc copy web2 server2web2 --verbose --refresh --mode=push

lxc config show server2:web2
architecture: x86_64
config:
image.architecture: amd64
image.description: ubuntu 18.04 LTS amd64 (release) (20210415)
image.label: release
image.os: ubuntu
image.release: bionic
image.serial: “20210415”
image.type: squashfs
image.version: “18.04”
volatile.base_image: cc14c502ccc1fe07f322649d9610fb240ed95866db72605afa33010cadfe9c22
volatile.eth0.hwaddr: 00:16:3e:e7:f7:20
volatile.idmap.base: “0”
volatile.idmap.current: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.idmap.next: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.last_state.idmap: ‘[{“Isuid”:true,“Isgid”:false,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000},{“Isuid”:false,“Isgid”:true,“Hostid”:1000000,“Nsid”:0,“Maprange”:1000000000}]’
volatile.uuid: 8ed67416-02c2-4ce0-b2c8-23e40ece7f8d
devices: {}
ephemeral: false
profiles:

default
stateful: false
description: “”
lxc network show server2:lxdbr0
config:
ipv4.address: 10.94.250.1/24
ipv4.nat: “true”
ipv6.address: fd42:3bd5:1f49:ee2e::1/64
ipv6.nat: “true”
description: “”
name: lxdbr0
type: bridge
used_by:

/1.0/instances/haproxy

/1.0/instances/web1

/1.0/instances/web2

/1.0/profiles/default

/1.0/profiles/haprofile
managed: true
status: Created
locations:

none

lxc start server2:web2
lxc exec server2:haproxy – bash
root@haproxy:~# ping web2.lxd
PING web2.lxd (10.94.250.225) 56(84) bytes of data.
From haproxy.lxd (10.94.250.219) icmp_seq=1 Destination Host Unreachable
From haproxy.lxd (10.94.250.219) icmp_seq=2 Destination Host Unreachable
From haproxy.lxd (10.94.250.219) icmp_seq=3 Destination Host Unreachable
^C
— web2.lxd ping statistics —
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 4079ms
pipe 4
root@haproxy:~#

I did not check the ipv4 address for server2:web2 after the first time run (after copying first without refresh mode)

lxc info server2:web2
Name: web2
Status: RUNNING
Type: container
Architecture: x86_64
PID: 1620536
Created: 2021/08/17 18:12 UTC
Last Used: 2021/08/17 18:35 UTC

Resources:
Processes: 24
Disk usage:
root: 914.71MiB
CPU usage:
CPU usage (in seconds): 4
Memory usage:
Memory (current): 130.04MiB
Memory (peak): 176.66MiB
Network usage:
eth0:
Type: broadcast
State: UP
Host interface: vethfc0beeb0
MAC address: 00:16:3e:e7:f7:20
MTU: 1500
Bytes received: 1.77kB
Bytes sent: 2.12kB
Packets received: 17
Packets sent: 20
IP addresses:
inet: 10.94.250.188/24 (global)
inet6: fd42:3bd5:1f49:ee2e:216:3eff:fee7:f720/64 (global)
inet6: fe80::216:3eff:fee7:f720/64 (link)
lo:
Type: loopback
State: UP
MTU: 65536
Bytes received: 276B
Bytes sent: 276B
Packets received: 4
Packets sent: 4
IP addresses:
inet: 127.0.0.1/8 (local)
inet6: ::1/128 (local)

One more detail from the server2:haproxy it is running now all the time, and it is configured with the server2:haprofile to use static ipv4.address. (forwarding from a host device haproxy related incoming http traffic):

lxc profile show server2:default
config: {}
description: Default LXD profile
devices:
eth0:
name: eth0
nictype: bridged
parent: lxdbr0
type: nic
root:
path: /
pool: lxd-pool
type: disk
name: default
used_by:

/1.0/instances/haproxy

/1.0/instances/web1

/1.0/instances/web2

lxc profile show server2:haprofile
config: {}
description: Bridged networking LXD profile
devices:
eth0:
ipv4.address: 10.94.250.219
nictype: bridged
parent: lxdbr0
type: nic
name: haprofile
used_by:

/1.0/instances/haproxy

vsuojanen · August 17, 2021, 7:54pm

I did wait some minutes this time and after some time the ping to the host web2.lxd start to work.

lxc exec server2:haproxy-- bash
root@haproxy:~# ping web2.lxd
PING web2.lxd (10.94.250.188) 56(84) bytes of data.
64 bytes from web2.lxd (10.94.250.188): icmp_seq=1 ttl=64 time=0.089 ms
64 bytes from web2.lxd (10.94.250.188): icmp_seq=2 ttl=64 time=0.065 ms
^C
— web2.lxd ping statistics —
2 packets transmitted, 2 received, 0% packet loss, time 1010ms
rtt min/avg/max/mdev = 0.065/0.077/0.089/0.012 ms

what’s the ipv4/ipv6 related difference when using lxc copy without refresh mode ?
I have had this small issue getting the private connection work between the haproxy and web1/web2 now on 3-4 different hosts after using --refresh mode exactly like in the last example here with web2

Let me know if there’s need to have some specific syslog from destination host.

tomp · August 23, 2021, 9:36am

Hi,

Sorry for the delay in replying to this.

So I’ve had a chance now to investigate this and I believe I have an answer for you.

When performing a copy of an instance without the --refresh flag the LXD client lxc strips the volatile.eth0.hwaddr setting, whereas with the --refresh flag the lxc client will remove the volatile.eth0.hwaddr setting, causing a new one to be generated with a different MAC address.

This happens even if there is already an existing instance with a different MAC address.

E.g.

lxc copy c1 c2 --refresh # c2 will have same volatile.eth0.hwaddr as c1.
lxc delete c2
lxc copy c1 c2 # c2 will have different volatile.eth0.hwaddr than c1.
lxc copy c1 c2 --refresh # c2 will now have same volatile.eth0.hwaddr as c1.

The change of MAC address between the initial copy of lxc copy c1 c2 without the --refresh flag is probably what is causing the pause of communication until the target server detects the instance’s MAC has changed.

Assuming this is the desired behaviour (CC @stgraber), then as the two servers have private bridges on separate networks and using the same MAC address on both won’t be an issue, then you can just to --refresh from the first copy and ensure the MAC address won’t change.

stgraber · August 23, 2021, 7:35pm

Yeah, from what I remember, we have lxc copy strip volatile whereas lxc move doesn’t. The reason being that we don’t want to cause conflicts when an instance is duplicated.

I believe it’s intentional that --refresh gets us the same behavior as lxc move as its goal is to perfectly replicate an instance as used for backups and the like.

vsuojanen · August 30, 2021, 11:57am

Thank you,

lxc copy (without --refresh) has not caused problems (delays detecting MAC changed.)

I was expecting it somehow that the way I was using lxc copy --refresh was not ideal for starting the same instance (refreshed duplicate) in the destination server on separate network. I

Andrew_Wilson · May 2, 2022, 1:54pm

Hi all:

I know the original question is answered, but I have a different twist on the issue.

I locally copy -refresh an instance (c1) from machine A to container (c2) on machine B on the same network - as part of a plan to make a local high-availability service (if machine A goes down, I have machine B on standby). If c1 goes down, I want c2 to be hot and ready to kick-in with the last --refreshed copy it has from Machine A. Howver, I need the IP /MAC address to change on the copy --refresh’d c2 instance because my lan obviously can’t handle two identical devices with same MAC/IP.

To further complicate my issue, I am using a netplan bridged network br0 on both host machines which I assign to c1 (from host 1) and the same thing on c2 (br0: also configured on host 2). My bridge setting br0 on my local machines makes my containers c1 and c2 accessible from my lan, without having the macvlan restriction of not being able to talk to host). [Thanks Stéphane Graber for the YouTube video on this!].

What I want to do is to change/specify the IP (and presumably mac?) of the copy --refresh’d c2 instance so that I can launch my second high-availiblty ‘backup’ copy when I need it.

So far, I have been unable to “reset” the mac/IP of the c2 copy --refresh’d container - google and this forum hasn’t quite gotten me there. I tried creating a netplan config but it ignored my change (i.e. changed eth0: dhcp4: to false, and specified an addresses: [10.0.0.xx/24].

Is it even possible to do what I am doing, and if so can you point me please? I.e. how do I change the IP address of the copy --refreshed instance c2 so that it gets a new static IP from the lan from the host br0: ?

Thanks in advance!

Andrew

tomp · May 2, 2022, 2:07pm

When running lxc copy --refresh both the contents of the instance and its config will be replaced so you will have to make any changes after each lxc copy invocation.

You can indeed modify the network configuration files inside the container to use a static IP address, see the netplan configs for examples:

Andrew_Wilson · May 2, 2022, 2:39pm

** EDITED

So it’s me again, not lxd. My darn netplan file had an error, so it would not pick up the new config.

I am leaving my post here in case some other poor soul has to learn this the hard way (I spent HOURS on this). Go into your container and manually apply netplan and see if it kicks an error. If it does (as mine did), it won’t give you an IP - but it won’t be obvious.

THANK YOU tomp - this is closed again now. Sorry for wasting some of your time.

Andrew

###############################################

Thank you. This fails for me, let me show:

For completeness, this is the new 50-cloud-init.yaml:

network:
version: 2
ethernets:
eth0:
dhcp4: false
addresses: [10.0.0.100/24]

So now, let’s start up c2 with supposedly a new netplan config:

aw@host:~$ lxc start c2

(wait some time …)

As you can see, in my case, I get NO IP for the container.

I tried this on two seperate machines and I get the same result - if I try to insert a new netplan config, it doesn’t take effect, it just stops me getting any IP which seems…wrong?

For completeness, here’s my host machine netplan configuration - there is an equivalent one on the other machine I tried:

network:
version: 2
renderer: networkd
ethernets:
enp3s0f1:
dhcp4: false
dhcp6: false
optional: true
bridges:
br0:
interfaces: [enp3s0f1]
addresses: [10.0.0.8/24]
gateway4: 10.0.0.1
nameservers:
addresses: [10.0.0.1,1.1.1.1]
parameters:
stp: false
forward-delay: 0
dhcp4: no
dhcp6: no
optional: true

Thanks for any pointers.

Andrew