How to setup routed nic inside fedora 33 container?

Dakado · January 26, 2021, 7:05pm

I have LXD container running the official fedora 33 image, and I am trying to add a routed nictype to it. I have LXD installed on Debian 10 machine via snap.

I have successfully setup routed nics on Debian, Devuan and CentoS however the behavior on fedora container is different.

After executing:
lxc config device add c1 eth0 nic nictype=routed parent=eth0 ipv4.address=$ip

lxc list command shows that the container does not have any IPv4 at all

lxc list
+------+---------+------+------+-----------+-----------+
| NAME |  STATE  | IPV4 | IPV6 |   TYPE    | SNAPSHOTS |
+------+---------+------+------+-----------+-----------+
| c1   | RUNNING |      |      | CONTAINER | 2         |
+------+---------+------+------+-----------+-----------+

eth0 interface appears disconnected inside the container:

nmcli device
DEVICE  TYPE      STATE         CONNECTION
eth0    ethernet  disconnected  --
lo      loopback  unmanaged     --

and when I try to change the IP address of that interface it says:

nmcli connection modify eth0 ipv4.addresses $ip
Error: unknown connection 'eth0'.

ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ca:1b:e8:5f:34:0b brd ff:ff:ff:ff:ff:ff link-netnsid 0

Any ideas what am I doing wrong ?

Thanks.

simos · January 26, 2021, 7:08pm

Hi!

(I edited your post for formatting per How to best ask questions on this discussion forum).

I think the issue is with nictype=routed parent=eth0. That is, it says that the host has an interface called eth0. Most hosts have interfaces named something like enp3s0. Can you verify that the network interface on the host is indeed eth0?

Dakado · January 26, 2021, 7:10pm

Hi, yes, the interface on the host is indeed eth0, since it is old debian 7 upgraded as the time went to debian 10, so the interface name did not changed.

As I stated above, using this method I was able to set routed nic on my debian and centos containers without any problem. But fedora somehow refuses this method.

simos · January 26, 2021, 7:26pm

You manage to get routed to work without a LXD profile with Debian, Devuan and CentOS? When you use routed, you need to touch a bit the networking configuration of the container so that works fine once it boots up. Because with routed, the container does not receive any network configuration over DHCP.

See the discussion here for some examples, Domain resolution in Debian 10 container with routed NIC

Dakado · January 26, 2021, 8:11pm

In debian I had to set the static ip inside /etc/networking/interfaces, but even before that lxc list already showed that the container has the given routed IP. However in fedora I cannot even execute a command to set the static IP because the interface somehow does not event exists (see in my first post).

tomp · January 26, 2021, 8:15pm

There may be a udev issue in the image we saw similar issues with centos 8, network manager is rather picky up which devices it will manage. Will take a look.

simos · January 26, 2021, 8:24pm

I tried using a profile per https://blog.simos.info/how-to-get-lxd-containers-get-ip-from-the-lan-with-routed-network/ and

$ lxc launch images:fedora/33/cloud myfedora --profile default --profile routed
Creating myfedora
Starting myfedora 
$ lxc shell myfedora
[root@myfedora ~]# ifconfig 
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[root@myfedora ~]# ifconfig eth0 up
[root@myfedora ~]# ifconfig 
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.201  netmask 255.255.255.255  broadcast 255.255.255.255
        inet6 fe80::e495:5eff:fecc:8e2a  prefixlen 64  scopeid 0x20<link>
        ether e6:95:5e:a0:22:5e  txqueuelen 1000  (Ethernet)
        RX packets 11  bytes 1123 (1.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 19  bytes 1494 (1.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[root@fedora ~]# nmcli 
eth0: connected (externally) to eth0
	"eth0"
	ethernet (veth), E6:95:5E:A0:22:5E, sw, mtu 1500
	inet4 192.168.1.201/32
	inet6 fe80::e495:5eff:fea0:832a/64
	route6 ff00::/8
	route6 fe80::/64

lo: unmanaged
	"lo"
	loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.
[root@myfedora ~]#

Requires more investigation. DNS requires to configure systemd-resolved (file /etc/systemd/resolved.conf).

Dakado · January 26, 2021, 9:29pm

So I did this, and ifconfig does not even work for me, so I am kinda confused looks like I have somehow a different version of fedora.

lxc init images:fedora/33 c1
lxc config device add c1 eth0 nic nictype=routed parent=eth0 ipv4.address=$ip
lxc start c1
lxc shell c1
ifconfig
-bash: ifconfig: command not found

simos · January 26, 2021, 9:40pm

I used the image images:fedora/33/cloud instead, which is like the images:fedora/33 but has cloud-init preinstalled and configured for you. I suppose the /cloud variant has ifconfig preinstalled.

Admittedly, you do not need to use ifconfig. I just used it for demonstration purposes. Newer versions of Linux distributions have by default the ip package. Therefore, per this guide, the equivalent command is

ip link set eth0 up

Dakado · January 26, 2021, 9:47pm

Alright that works, sorry I did not know that cloud version is different.

That command works, however in lxc list the container still does not appear to have an ipv4 address.

ip a output from inside the fedora container:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether da:5c:37:06:86:fb brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::d85c:37ff:fe06:86fb/64 scope link
       valid_lft forever preferred_lft forever

simos · January 26, 2021, 10:02pm

Here is my try. Note that I am using LXD profiles for this.

$ lxc launch images:fedora/33/cloud fedora --profile default --profile routed_fedora
Creating fedora
Starting fedora
$ lxc list fedora
+--------+---------+----------------------+------+-----------+-----------+
|  NAME  |  STATE  |         IPV4         | IPV6 |   TYPE    | SNAPSHOTS |
+--------+---------+----------------------+------+-----------+-----------+
| fedora | RUNNING | 192.168.1.201 (eth0) |      | CONTAINER | 0         |
+--------+---------+----------------------+------+-----------+-----------+
$ lxc shell fedora
[root@fedora ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if40: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether d2:4d:3d:1d:55:be brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.1.201/32 brd 255.255.255.255 scope global eth0
       valid_lft forever preferred_lft forever
[root@fedora ~]#

As I said earlier, there is need for extra work to make this all work. It does not work yet.
First, you need to use a LXD profile with the appropriate configuration.
Then, you need to manually complete the setup of the networking in the container.
Finally, touch up the LXD profile again so that it performs automatically for you that manual configuration you just did earlier.

Dakado · January 27, 2021, 6:30pm

Here is what I changed to make it work:

I added --profile default argument
Changed to the cloud version of fedora

Now ip a finally returned the correct routed IP address and I can see the address in lxc list command as well.

Maybe you could try that with the default fedora package, maybe there is some catch why it does not work on that image.

EDIT: After using nmcli to configure network, everything works flawlessly.

nmcli device
nmcli connection modify "System eth0" ipv4.addresses $ip/32
nmcli connection modify "System eth0" ipv4.gateway 169.254.0.1
nmcli connection modify "System eth0" ipv4.dns 8.8.8.8
nmcli connection modify "System eth0" ipv4.method manual
nmcli connection down "System eth0"
nmcli connection up "System eth0"

simos · January 27, 2021, 6:07am

It is great that you managed to get it working manually. If you are comfortable with that, then all is good.

If you would like to automate the process, then you can use cloud-init to do so. For that read carefully https://blog.simos.info/how-to-get-lxd-containers-get-ip-from-the-lan-with-routed-network/
As I said earlier, you can insert the manual instructions in an LXD profile. The profile default is the default profile, and is applied anyway. When you want to add another profile, then you need to specify both the default and you additional profile.

tomp · January 27, 2021, 1:59pm

@Dakado

I found that the Fedora cloud-init wasn’t applying the network config from @simos blog post.
It may be that because the interface already has an address, the NetworkManager is leaving it alone (the connection in nmcli showed it as being configured externally).

However you can automate the steps you documented above using a cloud-init profile as follows:

lxc profile show routed
config:
  user.user-data: |
    #cloud-config
    bootcmd:
      - nmcli connection modify "System eth0" ipv4.addresses 192.168.1.201/32
      - nmcli connection modify "System eth0" ipv4.gateway 169.254.0.1
      - nmcli connection modify "System eth0" ipv4.dns 8.8.8.8
      - nmcli connection modify "System eth0" ipv4.method manual
      - nmcli connection down "System eth0"
      - nmcli connection up "System eth0"
description: Default LXD profile
devices:
  eth0:
    ipv4.address: 192.168.1.201
    name: eth0
    nictype: routed
    parent: enp3s0
    type: nic
name: routed

simos · January 27, 2021, 1:54pm

Thanks, I updated the post at https://blog.simos.info/how-to-get-lxd-containers-get-ip-from-the-lan-with-routed-network/ to include Fedora.