Two containers with same ip address if restored from same backup

LXD
1

I just did following and now two containers are running with same IP.

$ lxc backup test1 test1.tar.gz --instance-only
$ lxc import test1.tar.gz test2
$ lxc start test2
$ lxc list

+----------------+---------+-------------------+------+-----------+-----------+----------------+
|      NAME      |  STATE  |       IPV4        | IPV6 |   TYPE    | SNAPSHOTS |    LOCATION    |
+----------------+---------+-------------------+------+-----------+-----------+----------------+
| test1          | RUNNING | 240.0.2.61 (eth0) |      | CONTAINER | 0         | do-01          |
+----------------+---------+-------------------+------+-----------+-----------+----------------+
| test2          | RUNNING | 240.0.2.61 (eth0) |      | CONTAINER | 0         | do-01          |
+----------------+---------+-------------------+------+-----------+-----------+----------------+

$ lxc info test1

Ips:
  lo:   inet    127.0.0.1
  lo:   inet6   ::1
  eth0: inet    240.0.2.61     veth564bad03
  eth0: inet6   fe80::216:3eff:fed2:de0f        veth564bad03

$ lxc info test2
Ips:
  eth0: inet    240.0.2.61     veth34c79e21
  eth0: inet6   fe80::216:3eff:fed2:de0f        veth34c79e21
  lo:   inet    127.0.0.1
  lo:   inet6   ::1

Is this an accepted behaviour?

I thought dnsmasq will not assign same ip to two containers.

2

Recently discovered this issue myself
 Its because the export backs up the hardware address (mac address) of the container.

Since DHCP keys IP address from the hardware address they will both be assigned the same IP.

The fix is to unset the hardware address on clones of your import and then lxd will automaticly assign it a new hardware address.

If your like myself and rely on the default profile for the lxdbr0 entry
 You will have to unset ‘volatile.eth0.hwaddr’ which works but is bad practice as all the volatile config is internal and subject to change without notice.

So on my system I import the container
 if its running I stop it

then I run

lxc config unset <containername> volatile.eth0.hwaddr

3

Also, if you’re looking to create multiple instances from a backup, then you could also consider using lxc publish to publish the instance as an image instead of lxc export. This would mean that each instance created from that image (using lxc init <image name>) will get its own fresh config (including MAC address).

4

I feel the correct behaviour should be to assign new IP . It leads to all kinds of errors if two instances are running with same IP.

1 Like
5

When using lxc export you are creating a backup of a single instance, and the intended use-case for that is to restore a single instance (because it is a backup).

When restoring the backup all of the original config is restored, including the MAC address.
A DHCP server generates IP assignments based on the MAC address, so if the MAC address is the same then the same IP will be allocated. However even aside from that issue, if you have two active instances with the same MAC address you are going to see other network issues.

Also if we started altering MAC address config on backup restoration this would cause issues for people who have used static IP allocation or network security rules based on MAC address.

If you’re looking to create multiple instances from a source instance, then you need to change the MAC address as @Ozymandias described, or use the lxc publish mechanism to create an image that can be used as the basis of multiple instances (which LXD then treats differently from restoring a backup by removing any volatile config, such as MAC address, so that each instance generates its own MAC address, and thus own IP allocation in DHCP).

1 Like
6

Here’s an example of using lxc publish:

# Create your source container and add your own modifications as needed.
lxc launch images:ubuntu/focal c1
lxc exec c1 -- touch /root/helloworld.txt
lxc stop c1

# Publish under a friendly alias.
lxc publish c1 --alias myimage
lxc image ls myimage
+---------+--------------+--------+-------------------------------------+--------------+-----------+----------+------------------------------+
|  ALIAS  | FINGERPRINT  | PUBLIC |             DESCRIPTION             | ARCHITECTURE |   TYPE    |   SIZE   |         UPLOAD DATE          |
+---------+--------------+--------+-------------------------------------+--------------+-----------+----------+------------------------------+
| myimage | facac9b50efd | no     | Ubuntu focal amd64 (20210223_07:42) | x86_64       | CONTAINER | 133.30MB | Feb 25, 2021 at 9:51am (UTC) |
+---------+--------------+--------+-------------------------------------+--------------+-----------+----------+------------------------------+

# Create container from image:
lxc launch myimage c2

# Check the new container contains our modifications:
lxc exec c2 -- ls /root/helloworld.txt
/root/helloworld.txt

# Start c1 and check both get own IPs:
lxc start c1
lxc ls
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |         IPV4         |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| c1   | RUNNING | 10.107.121.10 (eth0) | fd42:bd51:7486:d7b5:216:3eff:fe35:e509 (eth0) | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
| c2   | RUNNING | 10.107.121.7 (eth0)  | fd42:bd51:7486:d7b5:216:3eff:fe48:63fa (eth0) | CONTAINER | 0         |
+------+---------+----------------------+-----------------------------------------------+-----------+-----------+
7

If you want to restore from backup, and the original instance exists on the machine and is already running, you should be stopping it anyway.

8

I am ok with the current behaviour and made necessary changes in my system for backup and restore.

Still the command syntax of lxc import implies that a new container is created from the backup. If it’s lxc restore it will than be correct to assume that container will be the same including name. If the old container exist lxc restore will fail (this will indicate to sys admin to either rename the old container or delete it, before restoring the backup). This will mean the container name should automatically come from backup.

In my view this will help avoid having same Mac or IP address which will result in difficult to diagnose errors.

9

lxc restore already exists, it’s the command used to restore a snapshot.

10

How about extending it to define the backup file as another option?

I haven’t used lxc restore to restore from snapshots. I always launched a new container from snapshot and then just point the load-balancer towards new container. This makes it easier to have immutable container based infrastructure, in case the new container has an issue can point load-balancer to old container.

11

I just ran into this issue as well, and had to Google this thread to see what was going on. I didn’t even realize that the machines had duplicate IP addresses until after running the import operation multiple times.

If the “intended use case” is to restore to the same VM, then why is there an option that allows you to specify a new VM name? Shouldn’t the user have some kind of warning that there’s conflicting MAC addresses on the same network?

It’s obvious to an experienced user what’s happening, but it’s not necessarily at top of mind until you realize the effects later on. Beginner users, who don’t have networking experience, would be utterly confused at this.

12

I agree this is confusing, please can you log a github issue.

LXD does already have duplicate MAC detection (which is the root cause of this issue, 2 NICs with the same MAC).

For example:

lxc launch ubuntu:jammy c1
lxc ls
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| NAME |  STATE  |        IPV4        |                     IPV6                      |   TYPE    | SNAPSHOTS |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+
| c1   | RUNNING | 10.21.203.8 (eth0) | fd42:ffdb:caff:baf7:216:3eff:fee2:d8fb (eth0) | CONTAINER | 0         |
+------+---------+--------------------+-----------------------------------------------+-----------+-----------+

lxc config get c1 volatile.eth0.hwaddr
00:16:3e:e2:d8:fb

lxc init ubuntu:jammy c2
lxc config set c2 volatile.eth0.hwaddr=00:16:3e:e2:d8:fb
lxc config get c2 volatile.eth0.hwaddr
00:16:3e:e2:d8:fb
lxc start c2
Error: Failed start validation for device "eth0": MAC address "00:16:3e:e2:d8:fb" already defined on another NIC

So I would expect lxc import to succeed and then the start of the instance fail with that sort of error, allowing you to fix the issue with lxc config edit.

1 Like
13

Agreed, the import operation should succeed. There should also be an immediate warning displayed to the user, indicating that a MAC address (and/or other internal LXD unique identifiers) conflict has been detected. The warning should recommend corrective action.

Issue filed here: https://github.com/lxc/lxd/issues/11907

1 Like