LXC4 no network

Hi!
I tried lxc4 but no network. What is the problem?
Environment:
lxc-4.0.6-2.fc34.x86_64

test container: fedora34

--------------------------------------------------------------------------

Distribution configuration

lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = x86_64

Container specific configuration

lxc.rootfs.path = dir:/var/lib/lxc/fedora34/rootfs
lxc.uts.name = fedora34

Network configuration

lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = …

--------------------------------------------------------------------------

The ip address does not help too:
lxc.net.0.ipv4.address = 10.0.3.100/24
lxc.net.0.ipv4.gateway = 10.0.3.1

2: eth0@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:5a:0e:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::216:3eff:fe5a:ea2/64 scope link
valid_lft forever preferred_lft forever

ping 8.8.8.8
ping: connect: Network is unreachable

lxc-net is running:
lxc-net.service - LXC network bridge setup
Loaded: loaded (/usr/lib/systemd/system/lxc-net.service; enabled; vendor preset: disabled)
Active: active (exited) since Sat 2021-05-08 08:43:32 CEST; 20min ago
Docs: man:lxc
Main PID: 5899 (code=exited, status=0/SUCCESS)
Tasks: 1 (limit: 6999)
Memory: 2.0M
CGroup: /system.slice/lxc-net.service
└─6043 dnsmasq --conf-file=/etc/lxc/dnsmasq.conf -u dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override >

máj 08 08:43:32 fedora-nb systemd[1]: Starting LXC network bridge setup…
máj 08 08:43:32 fedora-nb dnsmasq[6043]: started, version 2.85 cachesize 150
máj 08 08:43:32 fedora-nb dnsmasq[6043]: compile time options: IPv6 GNU-getopt DBus no-UBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth cryptohash DNSSEC loop-detect inotify dumpfile
máj 08 08:43:32 fedora-nb dnsmasq-dhcp[6043]: DHCP, IP range 10.0.3.2 – 10.0.3.254, lease time 1h
máj 08 08:43:32 fedora-nb dnsmasq-dhcp[6043]: DHCP, sockets bound exclusively to interface lxcbr0
máj 08 08:43:32 fedora-nb dnsmasq[6043]: reading /etc/resolv.conf
máj 08 08:43:32 fedora-nb dnsmasq[6043]: using nameserver 127.0.0.53#53
máj 08 08:43:32 fedora-nb dnsmasq[6043]: read /etc/hosts - 9 addresses
máj 08 08:43:32 fedora-nb systemd[1]: Finished LXC network bridge setup.

Why?

Try a Fedora 32 or CentOS 8 container. Networking currently appears to be borked on Fedora 33, Fedora 34, and CentOS 8 Stream containers.

I can confirm for Fedora 34, network is completely borked indeed. In my case, my issue come from the fact that /sys doesn’t seems to be mounted read-only inside the container (a systemd requirement as described here https://systemd.io/CONTAINER_INTERFACE). Therefore, there is udevd triggered and everything goes wrong.

I fixed it by adding the following configuration in my profiles to force /sys to be mounted RO :

config:
  raw.lxc: lxc.mount.auto=sys

As described in the documentation : sys:mixed (or sys): mount /sys as read-only but with /sys/devices/virtual/net writable.

Other link : https://github.com/systemd/systemd/issues/14431#issuecomment-585356412

I will open an issue as soon as I can.

In all distro same problem, no network.
In centos8 I can setup manually network, but no works.

# in container:
$ ip addr add 10.0.3.101/24 dev eth0
$ ip route add default via 10.0.3.1 dev eth0

$ cat /etc/sysconfig/network-scripts/ifcfg-eth0 
DEVICE=eth0
BOOTPROTO=dhcp
ONBOOT=yes
HOSTNAME=centos8
TYPE=Ethernet
MTU=
DHCP_HOSTNAME=centos8

$  cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 8.8.8.8

It has no name resolver! For example:

$ dnf update
CentOS Linux 8 - AppStream 0.0 B/s | 0 B 01:00
Errors during downloading metadata for repository ‘appstream’:

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=112 time=15.8 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=112 time=24.1 ms

$ ip route
default via 10.0.3.1 dev eth0
10.0.3.0/24 dev eth0 proto kernel scope link src 10.0.3.101

:open_mouth:

See the line after you typed dnf update? Where it says CentOS Linux 8 - AppStream? That means you are running CentOS 8 Stream. Networking is currently borked (not working) in CentOS 8 Stream containers. Normal CentOS 8 containers work fine and can be installed with: lxc launch images:centos/8 centos-8

I just spent a week trying to figure out how I screwed up a new LXD install, only to figure out I did nothing wrong and CentOS 8 Stream containers don’t currently work. My new test procedure after an LXD install is to run:

lxc image list images: | grep -v cloud | grep -Po '^\| \K[^ ]+(?=.+x86_64.+CONTAINER)' | sed 'h;s#[^[:alnum:]]#-#g;x;G;s/\n/ /;s/^ */images:/' | xargs -n2 lxc launch
lxc list
lxc stop --all

This will install all popular distros and show me which ones got IPs and which ones shutdown correctly. If many containers work correctly, then my LXD install is probably fine. This also lets me open bug reports for those that don’t get both IPv4 and IPv6 addresses and shut down correctly so they can be fixed before I need them.

If you have less than 32GB RAM, watch your utilization and stop the above before it completes if necessary. Check the ones that are running, shut them all down, and then rerun the first command and it will continue where it left off without restarting the ones you already tested.

To delete all containers: lxc delete $(lxc list -c n --format csv)

CentOS 8 containers can be upgraded to Stream and will continue to work as long as you don’t let NetworkManager be updated. Just add exclude=NetworkManager* to /etc/yum.conf before running:

dnf install centos-release-stream
dnf swap centos-{linux,stream}-repos
dnf distro-sync

in CentOS7 container same problem:
$ You just created a Centos 7 x86_64 (20210509_07:08) container.

No network:

$ ip addr add 10.0.3.101/24 dev eth0
$ ip route add default via 10.0.3.1 dev eth0
$ ip route
default via 10.0.3.1 dev eth0
10.0.3.0/24 dev eth0 proto kernel scope link src 10.0.3.101

$ cat /etc/resolv.conf
nameserver 8.8.8.8

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=112 time=16.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=112 time=17.1 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=112 time=33.4 ms

$ yum update
Loading mirror speeds from cached hostfile
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=stock error was
14: curl#6 - “Could not resolve host: mirrorlist.centos.org; Unknown error”

$ ping ns.google.com
ping: ns.google.com: Name or service is unknown

Sorry, CentOS 7 works fine for me. If no containers work correctly, sounds like a host problem.

It is LXC4 problem (settings, template…). LXC3 worked fine (but it had some problem). There is usually problem at new version (after upgrade).
Could anybody show a sample dnsmasq config?
/etc/lxc/dnsmasq.conf
/etc/dnsmasq.conf

Please can you show output of ip a and ip r inside the container?

Default no ip, no route.
Manualy it works except DNS.

# in container:
$ ip addr add 10.0.3.100/24 dev eth0
$ ip route add default via 10.0.3.1 dev eth0
$ ip route
default via 10.0.3.1 dev eth0 
10.0.3.0/24 dev eth0 proto kernel scope link src 10.0.3.100

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=17.3 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=18.5 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=23.0 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=45.2 ms
# in container:
$ vi /etc/systemd/resolved.conf
DNS=8.8.8.8

# container restart
$ cat /etc/resolv.conf
nameserver 8.8.8.8
search .

$ ping ns.google.com
ping: ns.google.com: Name or service in unknown

Can anybody help?
Fedora34 container:
$ ip a
2: eth0@if9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 00:16:3e:5a:0e:a2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.0.3.102/24 brd 10.0.3.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::216:3eff:fe5a:ea2/64 scope link
valid_lft forever preferred_lft forever

$ ip r
default via 10.0.3.1 dev eth0 proto static
10.0.3.0/24 dev eth0 proto kernel scope link src 10.0.3.102

$ systemd-resolve --status
Global
Protocols: LLMNR=resolve -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: uplink
Current DNS Server: 8.8.8.8
DNS Servers: 8.8.8.8

Link 2 (eth0)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 8.8.4.4
DNS Servers: 8.8.8.8 8.8.4.4

$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=112 time=24.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=112 time=23.4 ms

$ ping ns.google.com
ping: ns.google.com: Name or service is unknown

Why no DNS? :frowning_face:

It works! :slight_smile:
There was no masquerade in host.
$ firewall-cmd --zone=FedoraWorkstation --add-masquerade --permanent

Thanks!!