Failing to pass traffic from container to host

[hgb]

Hi. We have a setup where we want to run waydroid (lineageOS) in an LXC container. The host os is a linux-from-scratch-type distro on an embedded arm platform.

We have a script for starting the container:

$ cat /bin/container
#!/usr/bin/env sh

lxc_args="-P /etc/lxc -n waydroid"
rootfs="/run/twist_rootfs"
bridge="br0"

start() {
    ip link add name "$bridge" type bridge
    ip link set dev "$bridge" up
    ip addr add 192.168.10.1/24 brd + dev "$bridge"

    sysctl -w net.ipv4.ip_forward=1
    iptables -I INPUT -i "$bridge" -p udp --dport 67 -j ACCEPT
    iptables -I INPUT -i "$bridge" -p tcp --dport 67 -j ACCEPT
    iptables -I INPUT -i "$bridge" -p udp --dport 53 -j ACCEPT
    iptables -I FORWARD -i "$bridge" -j ACCEPT
    iptables -I FORWARD -o "$bridge" -j ACCEPT
    iptables -t nat -A POSTROUTING -s 192.168.10.0/24 -j MASQUERADE

    mkdir /run/waydroid
    dnsmasq -u nobody --strict-order --bind-interfaces --pid-file=/run/waydroid/dnsmasq.pid --listen-address 192.168.10.1 --dhcp-range 192.168.10.2,192.168.10.10 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface="$bridge" --dhcp-leasefile=/run/waydroid/dnsmasq."$bridge".leases --dhcp-authoritative

    mkdir -p $rootfs
    mkdir -p /mnt/base/twist-data
    mkdir -p /mnt/base/host-permissions
    mount /mnt/base/system.img $rootfs
    mount -o remount,ro /mnt/base/system.img $rootfs
    mount /mnt/base/vendor.img $rootfs/vendor
    mount -o remount,ro /mnt/base/vendor.img $rootfs/vendor
    mount -o bind /etc/lxc/waydroid.prop $rootfs/vendor/waydroid.prop
    chmod 777 -R /run/xdg
    chmod 777 -R /dev/ashmem
    chmod 666 -R /dev/binder
    chmod 666 -R /dev/hwbinder
    chmod 666 -R /dev/vndbinder
    chmod 777 -R /dev/dri
    chmod 777 -R /dev/fb0
    chmod 777 -R /dev/video0
    chmod 777 -R /dev/video1
    chmod 777 -R /dev/video2
    chmod 777 -R /dev/video3
    lxc-start  -l debug -o /root/waydroid.log $lxc_args -- /init
}

stop() {
    lxc-stop $lxc_args -k
    umount $rootfs/vendor/waydroid.prop
    umount $rootfs/vendor
    umount $rootfs

    kill -9 $(cat /run/waydroid/dnsmasq.pid)
    rm -rf /run/waydroid

    iptables -t nat -D POSTROUTING -s 192.168.10.0/24 -j MASQUERADE
    iptables -D FORWARD -o "$bridge" -j ACCEPT
    iptables -D FORWARD -i "$bridge" -j ACCEPT
    iptables -D INPUT -i "$bridge" -p udp --dport 53 -j ACCEPT
    iptables -D INPUT -i "$bridge" -p tcp --dport 67 -j ACCEPT
    iptables -D INPUT -i "$bridge" -p udp --dport 67 -j ACCEPT

    sysctl -w net.ipv4.ip_forward=0

    ip link del dev "$bridge"
}

shell() {
    lxc-attach $lxc_args sh
}

logcat() {
    lxc-attach $lxc_args logcat
}

top() {
    lxc-top $lxc_args
}

info() {
    lxc-info $lxc_args
}

case "$1" in
    start)
        start
        ;;
    stop)
        stop
        ;;
    shell)
        shell
        ;;
    logcat)
        logcat
        ;;
    top)
        top
        ;;
    info)
        info
        ;;
    *)
        echo "Usage: $0 {start|stop|shell|logcat|top|info}"
        exit 1
        ;;
esac

If I run container start, then attach and either set up a manual IP address on 192.168.10.0/24 or use dhclient to get one (which works), I get a nice address I can ping from the network namespace, but I can’t reach it from the outside.

The host side has this:

5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether fe:c6:34:10:a7:c0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.1/24 brd 192.168.10.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::30c6:8bff:fe24:476c/64 scope link 
       valid_lft forever preferred_lft forever
6: vethB0ETE3@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether fe:c6:34:10:a7:c0 brd ff:ff:ff:ff:ff:ff link-netns ns1
    inet6 fe80::fcc6:34ff:fe10:a7c0/64 scope link 
       valid_lft forever preferred_lft forever

and the container side this:

[voctavio-2:~] $ ip netns exec ns1 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:f9:d3:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.10.4/24 brd 192.168.10.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fef9:d303/64 scope link 
       valid_lft forever preferred_lft forever

When I ping or try to add a default route:

[voctavio-2:~] $ ip netns exec ns1 ping 192.168.10.1
PING 192.168.10.1 (192.168.10.1): 56 data bytes
ping: sendto: Network is unreachable
[voctavio-2:~] $ ip netns exec ns1 ip route add default via 192.168.10.1
RTNETLINK answers: Network is unreachable

If I manually create a new network namespace with a new veth pair and attach it to the same bridge, it works as I would expect.

My config files:

[voctavio-2:~] $ cat /etc/lxc/default.conf 
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
[voctavio-2:~] $ cat /etc/lxc/waydroid.prop 
debug.stagefright.ccodec=0
ro.hardware.gralloc=default
ro.hardware.egl=swiftshader
ro.hardware.camera=v4l2
ro.opengles.version=196608
waydroid.system_ota=https://ota.waydro.id/system/lineage/waydroid_x86_64/VANILLA.json
waydroid.vendor_ota=https://ota.waydro.id/vendor/waydroid_x86_64/MAINLINE.json
waydroid.tools_version=1.2.0
ro.vndk.lite=true
ro.product.waydroid.model=Jetson-AGX
waydroid.host.user=root
waydroid.host.uid=1000
waydroid.host.gid=1000
waydroid.xdg_runtime_dir=/run/xdg
waydroid.pulse_runtime_path=/run/xdg/pulse
waydroid.wayland_display=wayland-0
waydroid.stub_sensors_hal=1
[voctavio-2:~] $ cat /etc/lxc/waydroid/config
# Twist LXC Config

lxc.rootfs.path = /run/twist_rootfs
lxc.uts.name = waydroid
lxc.arch = arm64
lxc.autodev = 0
# lxc.autodev.tmpfs.size = 25000000
lxc.apparmor.profile = unconfined

lxc.init.cmd = /init

lxc.mount.auto = cgroup:ro sys:ro proc

lxc.net.0.type = veth
lxc.net.0.flags = up
lxc.net.0.link = br0
lxc.net.0.name = eth0
lxc.net.0.hwaddr = 00:16:3e:f9:d3:03
lxc.net.0.mtu = 1500

lxc.console.path = none

lxc.include = /etc/lxc/waydroid/config_nodes

lxc.hook.post-stop = /dev/null
[voctavio-2:~] $ cat /etc/lxc/waydroid/config_nodes 
lxc.mount.entry = tmpfs dev tmpfs nosuid 0 0
lxc.mount.entry = /dev/zero dev/zero none bind,create=file,optional 0 0
lxc.mount.entry = /dev/null dev/null none bind,create=file,optional 0 0
lxc.mount.entry = /dev/full dev/full none bind,create=file,optional 0 0
lxc.mount.entry = /dev/ashmem dev/ashmem none bind,create=file,optional 0 0
lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,optional 0 0
lxc.mount.entry = /dev/char dev/char none bind,create=dir,optional 0 0
lxc.mount.entry = /dev/dri dev/dri none bind,create=dir,optional 0 0
lxc.mount.entry = /dev/fb0 dev/fb0 none bind,create=file,optional 0 0
lxc.mount.entry = /dev/binder dev/binder none bind,create=file,optional 0 0
lxc.mount.entry = /dev/vndbinder dev/vndbinder none bind,create=file,optional 0 0
lxc.mount.entry = /dev/hwbinder dev/hwbinder none bind,create=file,optional 0 0
lxc.mount.entry = none dev/pts devpts defaults,mode=644,ptmxmode=666,create=dir 0 0
lxc.mount.entry = /dev/uhid dev/uhid none bind,create=file,optional 0 0
lxc.mount.entry = tmpfs mnt tmpfs mode=0755,uid=0,gid=1000
lxc.mount.entry = /mnt/base/twist-data data none bind 0 0
lxc.mount.entry = /mnt/base/host-permissions vendor/etc/host-permissions none bind,optional 0 0
lxc.mount.entry = /run run none rbind,create=dir 0 0
lxc.mount.entry = /sys/kernel/debug sys/kernel/debug none rbind,create=dir,optional 0 0
lxc.mount.entry = tmpfs mnt_extra tmpfs nodev 0 0
lxc.mount.entry = tmpfs var tmpfs nodev 0 0
lxc.mount.entry = /var/run var/run none rbind,create=dir,optional 0 0
lxc.mount.entry = tmpfs tmp tmpfs nodev 0 0

Any ideas on how to get this bridge up?

[tomp (Thomas Parrott)]

Can you show output of ip netns exec ns1 ip r?

[tomp (Thomas Parrott)]

BTW have you tried using these settings in your container config?

lxc.net.[i].ipv4.address
lxc.net.[i].ipv4.gateway

See Linux Containers - LXC - Manpages - lxc.container.conf.5

[hgb]

@tomp

ip netns exec ns1 ip r

[voctavio-2:~] $ ip netns exec ns1 ip r
192.168.10.0/24 dev eth0 proto kernel scope link src 192.168.10.4 

[hgb]

Haven’t tried, can give it a shot.

[hgb]

@tomp

BTW have you tried using these settings in your container config?

lxc.net.[i].ipv4.address
lxc.net.[i].ipv4.gateway

Didn’t make a difference.

[tomp (Thomas Parrott)]

Does it work if you use lxc-attach and run ip route add manually?

[hgb]

No, it says “Network unreachable”. I get the same if I try to ping the gateway.

[hgb]

An interesting part is that getting IP via dhclient works, I just can’t get a route up, or IP(v4)-level connectivity.

[tomp (Thomas Parrott)]

What about:
ip route add default via 192.168.10.1 dev eth0

[hgb]

@tomp

ip route add default via 192.168.10.1 dev eth0

Same (Network unreachable)

[tomp (Thomas Parrott)]

I’m out of ideas im afraid.

Any ideas @brauner @stgraber could this be an android specific issue?

[stgraber (Stéphane Graber)]

That’s definitely pretty weird. It’s behaving as if the interface is DOWN even though it’s clearly not. Is there anything useful in the dmesg output?

[hgb]

I’ll try to dig tomorrow. The waydroid thing is really chatty about lots of things, but nothing I thought was obviously relevant while looking. We’re also looking to try to set up a minimal container on the same system to see if we can reproduce there.

[hgb]

We’ve had some more results from digging, and when we start a regular ubuntu container instead of waydroid, we get network, so the problem appears to be somewhere in whatever waydroid is doing. Thanks for helping out!

[hgb]

We seem to have fixed this through various fiddling with waydroid’s netd (actually android’s) daemon. Not that I’m really certain what the actual fix to the problem was… Anyway, thanks for your help.

[nirajgoit (Niraj Kumar Goit)]

Hello @hgb

We are really struggling with similar issue as yours.

Unable to ping container running in bridge mode from external PC

In a custom linux platform, I am running waydroid in bridge mode using LXC container. However, I am unable to ping to container from another PC in the same lan. Details of setup: External PC: 192.168.0.178 Linux PC (host): 192.168.0.11 Bridge interface: 192.168.10.1 Container: 192.168.10.13 Any ideas, why I am unable to ping container ‘192.168.10.13’ from external PC ‘192.168.0.178’. Getting error as ‘Destination unreachable’.

We wish to know how you have fixed the routing issue in LXC waydroid .
We are not able to ping/connect to waydroid container IP. Our host PC and external PC is in same LAN. Also, not able to ping external PC from inside the container.

We would really appreciate if you could help us with it.

[hgb]

We would really appreciate if you could help us with it.

I’m sorry, wish I could help, but we actually never figured out what we did to solve our problem. We’ve had issues later coming and going, but they have been related to starting the container before the host was ready (for example before the interface was set up, or before the dhcp server started).

[Baumbart]

I had the exact same issue, when using a custom kernel config.
When looking at logcat, i found that netd was complaining about something IPV6 related, even though I already had CONFIG_IPV6 enabled.
I then enabled every config option I could find under > Networking support > Networking options > The IPv6 protocol and it worked afterwards.