Failing to pass traffic from container to host

Hi. We have a setup where we want to run waydroid (lineageOS) in an LXC container. The host os is a linux-from-scratch-type distro on an embedded arm platform.

We have a script for starting the container:

$ cat /bin/container
#!/usr/bin/env sh

lxc_args="-P /etc/lxc -n waydroid"
rootfs="/run/twist_rootfs"
bridge="br0"

start() {
    ip link add name "$bridge" type bridge
    ip link set dev "$bridge" up
    ip addr add 192.168.10.1/24 brd + dev "$bridge"

    sysctl -w net.ipv4.ip_forward=1
    iptables -I INPUT -i "$bridge" -p udp --dport 67 -j ACCEPT
    iptables -I INPUT -i "$bridge" -p tcp --dport 67 -j ACCEPT
    iptables -I INPUT -i "$bridge" -p udp --dport 53 -j ACCEPT
    iptables -I FORWARD -i "$bridge" -j ACCEPT
    iptables -I FORWARD -o "$bridge" -j ACCEPT
    iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -j MASQUERADE

    mkdir /run/waydroid
    dnsmasq -u nobody --strict-order --bind-interfaces --pid-file=/run/waydroid/dnsmasq.pid --listen-address 192.168.10.1 --dhcp-range 192.168.10.2,192.168.10.10 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface="$bridge" --dhcp-leasefile=/run/waydroid/dnsmasq."$bridge".leases --dhcp-authoritative

    mkdir -p $rootfs
    mkdir -p /mnt/base/twist-data
    mkdir -p /mnt/base/host-permissions
    mount /mnt/base/system.img $rootfs
    mount -o remount,ro /mnt/base/system.img $rootfs
    mount /mnt/base/vendor.img $rootfs/vendor
    mount -o remount,ro /mnt/base/vendor.img $rootfs/vendor
    mount -o bind /etc/lxc/waydroid.prop $rootfs/vendor/waydroid.prop
    chmod 777 -R /run/xdg
    chmod 777 -R /dev/ashmem
    chmod 666 -R /dev/binder
    chmod 666 -R /dev/hwbinder
    chmod 666 -R /dev/vndbinder
    chmod 777 -R /dev/dri
    chmod 777 -R /dev/fb0
    chmod 777 -R /dev/video0
    chmod 777 -R /dev/video1
    chmod 777 -R /dev/video2
    chmod 777 -R /dev/video3
    lxc-start  -l debug -o /root/waydroid.log $lxc_args -- /init
}

stop() {
    lxc-stop $lxc_args -k
    umount $rootfs/vendor/waydroid.prop
    umount $rootfs/vendor
    umount $rootfs

    kill -9 $(cat /run/waydroid/dnsmasq.pid)
    rm -rf /run/waydroid

    iptables -t nat -D POSTROUTING -s 192.168.10.0/24 -j MASQUERADE
    iptables -D FORWARD -o "$bridge" -j ACCEPT
    iptables -D FORWARD -i "$bridge" -j ACCEPT
    iptables -D INPUT -i "$bridge" -p udp --dport 53 -j ACCEPT
    iptables -D INPUT -i "$bridge" -p tcp --dport 67 -j ACCEPT
    iptables -D INPUT -i "$bridge" -p udp --dport 67 -j ACCEPT

    sysctl -w net.ipv4.ip_forward=0

    ip link del dev "$bridge"
}

shell() {
    lxc-attach $lxc_args sh
}

logcat() {
    lxc-attach $lxc_args logcat
}

top() {
    lxc-top $lxc_args
}

info() {
    lxc-info $lxc_args
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    shell)
        shell
        ;;
    logcat)
        logcat
        ;;
    top)
        top
        ;;
    info)
        info
        ;;
    *)
        echo "Usage: $0 {start|stop|shell|logcat|top|info}"
        exit 1
        ;;
esac

If I run container start, then attach and either set up a manual IP address on 192.168.10.0/24 or use dhclient to get one (which works), I get a nice address I can ping from the network namespace, but I can’t reach it from the outside.

The host side has this:

5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:c6:34:10:a7:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.1/24 brd 192.168.10.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::30c6:8bff:fe24:476c/64 scope link 
       valid_lft forever preferred_lft forever
6: vethB0ETE3@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether fe:c6:34:10:a7:c0 brd ff:ff:ff:ff:ff:ff link-netns ns1
    inet6 fe80::fcc6:34ff:fe10:a7c0/64 scope link 
       valid_lft forever preferred_lft forever

and the container side this:

[voctavio-2:~] $ ip netns exec ns1 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:f9:d3:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.10.4/24 brd 192.168.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fef9:d303/64 scope link 
       valid_lft forever preferred_lft forever

When I ping or try to add a default route:

[voctavio-2:~] $ ip netns exec ns1 ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1): 56 data bytes
ping: sendto: Network is unreachable
[voctavio-2:~] $ ip netns exec ns1 ip route add default via 192.168.10.1
RTNETLINK answers: Network is unreachable

If I manually create a new network namespace with a new veth pair and attach it to the same bridge, it works as I would expect.

My config files:

[voctavio-2:~] $ cat /etc/lxc/default.conf 
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
[voctavio-2:~] $ cat /etc/lxc/waydroid.prop 
debug.stagefright.ccodec=0
ro.hardware.gralloc=default
ro.hardware.egl=swiftshader
ro.hardware.camera=v4l2
ro.opengles.version=196608
waydroid.system_ota=https://ota.waydro.id/system/lineage/waydroid_x86_64/VANILLA.json
waydroid.vendor_ota=https://ota.waydro.id/vendor/waydroid_x86_64/MAINLINE.json
waydroid.tools_version=1.2.0
ro.vndk.lite=true
ro.product.waydroid.model=Jetson-AGX
waydroid.host.user=root
waydroid.host.uid=1000
waydroid.host.gid=1000
waydroid.xdg_runtime_dir=/run/xdg
waydroid.pulse_runtime_path=/run/xdg/pulse
waydroid.wayland_display=wayland-0
waydroid.stub_sensors_hal=1
[voctavio-2:~] $ cat /etc/lxc/waydroid/config
# Twist LXC Config

lxc.rootfs.path = /run/twist_rootfs
lxc.uts.name = waydroid
lxc.arch = arm64
lxc.autodev = 0
# lxc.autodev.tmpfs.size = 25000000
lxc.apparmor.profile = unconfined

lxc.init.cmd = /init

lxc.mount.auto = cgroup:ro sys:ro proc

lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.name = eth0
lxc.net.0.hwaddr = 00:16:3e:f9:d3:03
lxc.net.0.mtu = 1500

lxc.console.path = none

lxc.include = /etc/lxc/waydroid/config_nodes

lxc.hook.post-stop = /dev/null
[voctavio-2:~] $ cat /etc/lxc/waydroid/config_nodes 
lxc.mount.entry = tmpfs dev tmpfs nosuid 0 0
lxc.mount.entry = /dev/zero dev/zero none bind,create=file,optional 0 0
lxc.mount.entry = /dev/null dev/null none bind,create=file,optional 0 0
lxc.mount.entry = /dev/full dev/full none bind,create=file,optional 0 0
lxc.mount.entry = /dev/ashmem dev/ashmem none bind,create=file,optional 0 0
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/char dev/char none bind,create=dir,optional 0 0
lxc.mount.entry = /dev/dri dev/dri none bind,create=dir,optional 0 0
lxc.mount.entry = /dev/fb0 dev/fb0 none bind,create=file,optional 0 0
lxc.mount.entry = /dev/binder dev/binder none bind,create=file,optional 0 0
lxc.mount.entry = /dev/vndbinder dev/vndbinder none bind,create=file,optional 0 0
lxc.mount.entry = /dev/hwbinder dev/hwbinder none bind,create=file,optional 0 0
lxc.mount.entry = none dev/pts devpts defaults,mode=644,ptmxmode=666,create=dir 0 0
lxc.mount.entry = /dev/uhid dev/uhid none bind,create=file,optional 0 0
lxc.mount.entry = tmpfs mnt tmpfs mode=0755,uid=0,gid=1000
lxc.mount.entry = /mnt/base/twist-data data none bind 0 0
lxc.mount.entry = /mnt/base/host-permissions vendor/etc/host-permissions none bind,optional 0 0
lxc.mount.entry = /run run none rbind,create=dir 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = tmpfs mnt_extra tmpfs nodev 0 0
lxc.mount.entry = tmpfs var tmpfs nodev 0 0
lxc.mount.entry = /var/run var/run none rbind,create=dir,optional 0 0
lxc.mount.entry = tmpfs tmp tmpfs nodev 0 0

Any ideas on how to get this bridge up?

Can you show output of ip netns exec ns1 ip r?

BTW have you tried using these settings in your container config?

lxc.net.[i].ipv4.address
lxc.net.[i].ipv4.gateway

See Linux Containers - LXC - Manpages - lxc.container.conf.5

[voctavio-2:~] $ ip netns exec ns1 ip r
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.4 

Haven’t tried, can give it a shot.

Didn’t make a difference.

Does it work if you use lxc-attach and run ip route add manually?

No, it says “Network unreachable”. I get the same if I try to ping the gateway.

An interesting part is that getting IP via dhclient works, I just can’t get a route up, or IP(v4)-level connectivity.

What about:
ip route add default via 192.168.10.1 dev eth0

Same (Network unreachable)

I’m out of ideas im afraid.

Any ideas @brauner @stgraber could this be an android specific issue?

That’s definitely pretty weird. It’s behaving as if the interface is DOWN even though it’s clearly not. Is there anything useful in the dmesg output?

I’ll try to dig tomorrow. The waydroid thing is really chatty about lots of things, but nothing I thought was obviously relevant while looking. We’re also looking to try to set up a minimal container on the same system to see if we can reproduce there.

We’ve had some more results from digging, and when we start a regular ubuntu container instead of waydroid, we get network, so the problem appears to be somewhere in whatever waydroid is doing. Thanks for helping out!

We seem to have fixed this through various fiddling with waydroid’s netd (actually android’s) daemon. Not that I’m really certain what the actual fix to the problem was… Anyway, thanks for your help.

1 Like